Max level shown:
12026-03-10T20:08:01.158ZDEBGtest_retry_backoff (init_with_steps): Running step start_crdb
22026-03-10T20:08:01.165ZDEBGtest_retry_backoff: Starting CRDB
32026-03-10T20:08:01.223ZINFOtest_retry_backoff: cockroach temporary directory: /var/tmp/omicron_tmp/.tmpgPwxcY
42026-03-10T20:08:01.223ZINFOtest_retry_backoff: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/a3c652d352c94636ee55aed34a080f27929619368ace6d1afadc8595a0b32f2f.tar) to storage directory (/var/tmp/omicron_tmp/.tmpgPwxcY/data)
52026-03-10T20:08:01.243ZINFOtest_retry_backoff: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmpgPwxcY/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmpgPwxcY/listen-url --max-sql-memory 256MiB
62026-03-10T20:08:01.243ZINFOtest_retry_backoff: cockroach environment: BUILDOMAT_JOB_ID=01KKCFSACVGWZBGX541J2KB207 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.91.1-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/a3c652d352c94636ee55aed34a080f27929619368ace6d1afadc8595a0b32f2f.tar GITHUB_BRANCH=alan/test-wip-discard GITHUB_REF=refs/heads/alan/test-wip-discard GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=c1b514da82f27c57b8ea415a61a4bc71b9181f9a GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-b2991dcfe0032b19/out:/work/oxidecomputer/omicron/target/debug/build/blake3-df20c6b18d3956e5/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-8b0f050bf31de6f5/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-fc1bb2730a9a80d7/out/build:/work/oxidecomputer/omicron/target/debug/build/psm-e3b3fd8e545b20fd/out:/work/oxidecomputer/omicron/target/debug/build/ring-882060e0ad2a422c/out:/work/oxidecomputer/omicron/target/debug/build/tofino-d064271f908edbf6/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-54b41a4eb78e9d96/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.91.1-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.91.1-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_ATTEMPT=1 NEXTEST_ATTEMPT_ID=06f00386-4a1b-45ce-bf0d-c0b32e4bdb09:omicron-nexus::test_all$integration_tests::webhooks::test_retry_backoff NEXTEST_BINARY_ID=omicron-nexus::test_all NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_BIN_EXE_schema_updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-b2991dcfe0032b19/out:/work/oxidecomputer/omicron/target/debug/build/blake3-df20c6b18d3956e5/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-8b0f050bf31de6f5/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-fc1bb2730a9a80d7/out/build:/work/oxidecomputer/omicron/target/debug/build/psm-e3b3fd8e545b20fd/out:/work/oxidecomputer/omicron/target/debug/build/ring-882060e0ad2a422c/out:/work/oxidecomputer/omicron/target/debug/build/tofino-d064271f908edbf6/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-54b41a4eb78e9d96/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.91.1-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.91.1-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=06f00386-4a1b-45ce-bf0d-c0b32e4bdb09 NEXTEST_RUN_MODE=test NEXTEST_STATE_DIR=/tmp/nextest-state.OFaWCs NEXTEST_STRESS_CURRENT=none NEXTEST_STRESS_TOTAL=none NEXTEST_TEST_GLOBAL_SLOT=5 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none NEXTEST_TEST_NAME=integration_tests::webhooks::test_retry_backoff NEXTEST_TEST_PHASE=run NEXTEST_TOTAL_ATTEMPTS=1 OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-3b5642a0768e3b39/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=--cfg tokio_unstable -D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.91.1-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime
72026-03-10T20:08:02.388ZINFOtest_retry_backoff: cockroach pid: 4784
82026-03-10T20:08:02.388ZINFOtest_retry_backoff: cockroach listen URL: postgresql://root@[::1]:42738/omicron?sslmode=disable
92026-03-10T20:08:02.511ZDEBGtest_retry_backoff (init_with_steps): Running step start_clickhouse
102026-03-10T20:08:02.511ZDEBGtest_retry_backoff: Starting Clickhouse
112026-03-10T20:08:03.016ZDEBGtest_retry_backoff (init_with_steps): Running step start_internal_dns
122026-03-10T20:08:03.016ZINFOtest_retry_backoff (store): opening sled database path = /var/tmp/omicron_tmp/.tmp4YPRj1
132026-03-10T20:08:03.076ZINFOtest_retry_backoff (store): pruning trees for generations newer than 0
142026-03-10T20:08:03.076ZINFOtest_retry_backoff (store): pruning trees for generations older than 0 keep = 3
152026-03-10T20:08:03.077ZINFOtest_retry_backoff (dns): DNS server bound to address kind = dns local_address = [::1]:49304
162026-03-10T20:08:03.077ZDEBGtest_retry_backoff (http): registered endpoint kind = dns local_addr = [::1]:64860 method = GET path = /config versions = all starting from 2.0.0
172026-03-10T20:08:03.077ZDEBGtest_retry_backoff (http): registered endpoint kind = dns local_addr = [::1]:64860 method = GET path = /config versions = from 1.0.0 to 2.0.0
182026-03-10T20:08:03.077ZDEBGtest_retry_backoff (http): registered endpoint kind = dns local_addr = [::1]:64860 method = PUT path = /config versions = all starting from 2.0.0
192026-03-10T20:08:03.077ZDEBGtest_retry_backoff (http): registered endpoint kind = dns local_addr = [::1]:64860 method = PUT path = /config versions = from 1.0.0 to 2.0.0
202026-03-10T20:08:03.077ZINFOtest_retry_backoff (http): listening kind = dns local_addr = [::1]:64860
212026-03-10T20:08:03.078ZDEBGtest_retry_backoff (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:64860
222026-03-10T20:08:03.078ZDEBGtest_retry_backoff (init_with_steps): Running step start_external_dns
232026-03-10T20:08:03.078ZINFOtest_retry_backoff (store): opening sled database path = /var/tmp/omicron_tmp/.tmpFE9NWR
242026-03-10T20:08:03.084ZINFOtest_retry_backoff (store): pruning trees for generations newer than 0
252026-03-10T20:08:03.084ZINFOtest_retry_backoff (store): pruning trees for generations older than 0 keep = 3
262026-03-10T20:08:03.084ZINFOtest_retry_backoff (dns): DNS server bound to address kind = dns local_address = [::1]:40416
272026-03-10T20:08:03.084ZDEBGtest_retry_backoff (http): registered endpoint kind = dns local_addr = [::1]:61901 method = GET path = /config versions = all starting from 2.0.0
282026-03-10T20:08:03.084ZDEBGtest_retry_backoff (http): registered endpoint kind = dns local_addr = [::1]:61901 method = GET path = /config versions = from 1.0.0 to 2.0.0
292026-03-10T20:08:03.084ZDEBGtest_retry_backoff (http): registered endpoint kind = dns local_addr = [::1]:61901 method = PUT path = /config versions = all starting from 2.0.0
302026-03-10T20:08:03.084ZDEBGtest_retry_backoff (http): registered endpoint kind = dns local_addr = [::1]:61901 method = PUT path = /config versions = from 1.0.0 to 2.0.0
312026-03-10T20:08:03.084ZINFOtest_retry_backoff (http): listening kind = dns local_addr = [::1]:61901
322026-03-10T20:08:03.084ZDEBGtest_retry_backoff (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:61901
332026-03-10T20:08:03.085ZDEBGtest_retry_backoff (init_with_steps): Running step start_nexus_internal
342026-03-10T20:08:03.085ZDEBGtest_retry_backoff: Starting Nexus (internal API)
352026-03-10T20:08:03.085ZINFOtest_retry_backoff: setting up nexus server
362026-03-10T20:08:03.087ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Action
372026-03-10T20:08:03.087ZINFOtest_retry_backoff (ServerContext): registering Oso class class = AuditLog
382026-03-10T20:08:03.087ZINFOtest_retry_backoff (ServerContext): registering Oso class class = AnyActor
392026-03-10T20:08:03.087ZINFOtest_retry_backoff (ServerContext): registering Oso class class = AuthenticatedActor
402026-03-10T20:08:03.087ZINFOtest_retry_backoff (ServerContext): registering Oso class class = BlueprintConfig
412026-03-10T20:08:03.087ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Database
422026-03-10T20:08:03.087ZINFOtest_retry_backoff (ServerContext): registering Oso class class = DnsConfig
432026-03-10T20:08:03.087ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Fleet
442026-03-10T20:08:03.087ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Inventory
452026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = IpPoolList
462026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = VpcList
472026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = ConsoleSessionList
482026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = DeviceAuthRequestList
492026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = QuiesceState
502026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SiloCertificateList
512026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SiloGroupList
522026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SiloIdentityProviderList
532026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SiloUserList
542026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SiloUserSessionList
552026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SiloUserTokenList
562026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SubnetPoolList
572026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = UpdateTrustRootList
582026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = TargetReleaseConfig
592026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = AlertClassList
602026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = ScimClientBearerTokenList
612026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = MulticastGroupList
622026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Project
632026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Disk
642026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Snapshot
652026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = ProjectImage
662026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = AffinityGroup
672026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = AntiAffinityGroup
682026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Instance
692026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = IpPool
702026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = InstanceNetworkInterface
712026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Vpc
722026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = VpcRouter
732026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = InternetGateway
742026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = InternetGatewayIpPool
752026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = InternetGatewayIpAddress
762026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = RouterRoute
772026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = VpcSubnet
782026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = FloatingIp
792026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = ExternalSubnet
802026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Image
812026-03-10T20:08:03.088ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SiloImage
822026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = AddressLot
832026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Blueprint
842026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = LoopbackAddress
852026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Certificate
862026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = ConsoleSession
872026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = DeviceAuthRequest
882026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = DeviceAccessToken
892026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = PhysicalDisk
902026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Rack
912026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SshKey
922026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Silo
932026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SiloUser
942026-03-10T20:08:03.089ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SiloGroup
952026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SupportBundle
962026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = IdentityProvider
972026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SamlIdentityProvider
982026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Sled
992026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = SubnetPool
1002026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = TufRepo
1012026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = TufArtifact
1022026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = TufTrustRoot
1032026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Alert
1042026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = AlertReceiver
1052026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = WebhookSecret
1062026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Zpool
1072026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = Service
1082026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = UserBuiltin
1092026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = ScimClientBearerToken
1102026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): registering Oso class class = MulticastGroup
1112026-03-10T20:08:03.090ZINFOtest_retry_backoff (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations; grants project.admin on all projects)\n# - silo.limited-collaborator (grants project.limited-collaborator on all projects)\n# - silo.viewer (can read most resources within the Silo; grants project.viewer)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project, including networking)\n# - project.limited-collaborator (can manage compute resources, but not networking resources)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\n# Note that certain Actors may not ever have any roles assigned to them, like\n# SCIM Actors.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "limited-collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "limited-collaborator";\n\t"limited-collaborator" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif actor.is_user and silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "limited-collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t# Role hierarchy: admin > collaborator > limited-collaborator > viewer\n\t#\n\t# The "limited-collaborator" role can create/modify non-networking\n\t# resources (instances, disks, etc.) but cannot create/modify networking\n\t# infrastructure (VPCs, subnets, routers, internet gateways).\n\t# See nexus/authz-macros for InProjectLimited vs InProjectFull.\n\t"viewer" if "limited-collaborator";\n\t"limited-collaborator" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "limited-collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"limited-collaborator" if "limited-collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if actor.is_user and silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n}\n# We want to allow the user to modify the ssh key but disallow a SCIM IdP token\n# from doing the same.\nhas_permission(actor: AuthenticatedActor, "modify", ssh_key: SshKey)\n\tif actor.is_user and has_permission(actor, "modify", ssh_key.silo_user);\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource ScimClientBearerToken {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n # necessary to authenticate SCIM actors\n\t"read" if "external-authenticator" on "parent_fleet";\n\n\t# Silo-level roles grant privileges for SCIM client tokens.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges for SCIM client tokens.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", scim_client_bearer_token: ScimClientBearerToken)\n\tif scim_client_bearer_token.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: ScimClientBearerToken)\n\tif collection.silo.fleet = fleet;\n\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the quiesce state of a particular Nexus instance.\n#\n# These authz checks must not require the database. We grant this directly to\n# callers of the internal API.\nresource QuiesceState {\n\tpermissions = [ "read", "modify" ];\n}\nhas_permission(USER_INTERNAL_API: AuthenticatedActor, "read", _q: QuiesceState);\nhas_permission(\n USER_INTERNAL_API: AuthenticatedActor,\n "modify",\n _q: QuiesceState\n);\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/update/trust-roots" in the API\nresource UpdateTrustRootList {\n\tpermissions = [ "list_children", "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"list_children" if "viewer" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: UpdateTrustRootList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif actor.is_user and silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for accessing "/v1/multicast-groups" in the API\n# Groups are created when the first instance joins and deleted when the last leaves.\nresource MulticastGroupList {\n\tpermissions = [ "list_children" ];\n\n\trelations = { parent_fleet: Fleet };\n\n\t# Fleet Viewers can list multicast groups\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", multicast_group_list: MulticastGroupList)\n\tif multicast_group_list.fleet = fleet;\n\n# Any authenticated user can list multicast groups in their fleet.\n# This enables silo users to discover groups for attaching instances,\n# without requiring the Fleet::Viewer role.\nhas_permission(actor: AuthenticatedActor, "list_children", multicast_group_list: MulticastGroupList)\n\tif actor.is_user and silo in actor.silo and silo.fleet = multicast_group_list.fleet;\n\n# MulticastGroup is a fleet-level discovery resource.\n# Join/leave authorization is gated by Instance::Modify, not the group itself.\nresource MulticastGroup {\n\tpermissions = [ "read", "list_children" ];\n\trelations = { parent_fleet: Fleet };\n}\nhas_relation(fleet: Fleet, "parent_fleet", multicast_group: MulticastGroup)\n\tif multicast_group.fleet = fleet;\n\n# Any authenticated user can read multicast groups in their fleet\nhas_permission(actor: AuthenticatedActor, "read", multicast_group: MulticastGroup)\n\tif actor.is_user and silo in actor.silo and silo.fleet = multicast_group.fleet;\n\n# Any authenticated user can list members of multicast groups in their fleet\nhas_permission(actor: AuthenticatedActor, "list_children", multicast_group: MulticastGroup)\n\tif actor.is_user and silo in actor.silo and silo.fleet = multicast_group.fleet;\n\n# Describes the policy for reading and writing the audit log\nresource AuditLog {\n\tpermissions = [\n\t "list_children", # retrieve audit log\n\t "create_child", # create audit log entry\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\n\t# Fleet viewers can read the audit log\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\n# Any actor should be able to write to the audit log because we need to be able\n# to write to the log from any request, authenticated or not. Audit log writes\n# are always a byproduct of other operations: there are no endpoints that allow\n# the user to write to the log deliberately. Note we use AuthenticatedActor\n# because we don't really mean unauthenticated -- in the case of login\n# operations, we use the external authenticator actor that creates the session\n# to authorize the audit log write.\nhas_permission(_actor: AuthenticatedActor, "create_child", _audit_log: AuditLog);\n\nhas_relation(fleet: Fleet, "parent_fleet", audit_log: AuditLog)\n\tif audit_log.fleet = fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Allow silo admins to delete and list user sessions\nresource SiloUserSessionList {\n permissions = [ "modify", "list_children" ];\n relations = { parent_silo: Silo };\n\n # A silo admin can modify (e.g., delete) a user's sessions.\n "modify" if "admin" on "parent_silo";\n\n # A silo admin can list a user's sessions.\n "list_children" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", authn_list: SiloUserSessionList)\n if authn_list.silo_user.silo = silo;\n\n# give users 'modify' and 'list_children' on their own sessions\nhas_permission(actor: AuthenticatedActor, "modify", authn_list: SiloUserSessionList)\n if actor.equals_silo_user(authn_list.silo_user);\nhas_permission(actor: AuthenticatedActor, "list_children", authn_list: SiloUserSessionList)\n if actor.equals_silo_user(authn_list.silo_user);\n\n# Allow silo admins to delete and list user access tokens\nresource SiloUserTokenList {\n permissions = [ "modify", "list_children" ];\n relations = { parent_silo: Silo };\n\n # A silo admin can modify (e.g., delete) a user's tokens.\n "modify" if "admin" on "parent_silo";\n\n # A silo admin can list a user's tokens.\n "list_children" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", authn_list: SiloUserTokenList)\n if authn_list.silo_user.silo = silo;\n\n# give users 'modify' and 'list_children' on their own tokens\nhas_permission(actor: AuthenticatedActor, "modify", authn_list: SiloUserTokenList)\n if actor.equals_silo_user(authn_list.silo_user);\nhas_permission(actor: AuthenticatedActor, "list_children", authn_list: SiloUserTokenList)\n if actor.equals_silo_user(authn_list.silo_user);\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# Grant SCIM IdP actors the permissions they need on users.\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if actor.is_scim_idp and silo_user.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "create_child", silo_user_list: SiloUserList)\n\tif actor.is_scim_idp and silo_user_list.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "modify", silo_user: SiloUser)\n\tif actor.is_scim_idp and silo_user.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "list_children", silo_user_list: SiloUserList)\n if actor.is_scim_idp and silo_user_list.silo in actor.silo;\n\n# Describes the policy for creating and managing Silo groups (mostly intended\n# for API-managed groups)\nresource SiloGroupList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the groups in the\n\t# Silo) can see the groups in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's groups. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloGroupList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloGroupList)\n\tif collection.silo.fleet = fleet;\n\n# Grant SCIM IdP actors the permissions they need on groups.\nhas_permission(actor: AuthenticatedActor, "read", silo_group: SiloGroup)\n if actor.is_scim_idp and silo_group.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "create_child", silo_group_list: SiloGroupList)\n\tif actor.is_scim_idp and silo_group_list.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "modify", silo_group: SiloGroup)\n\tif actor.is_scim_idp and silo_group.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "list_children", silo_group_list: SiloGroupList)\n if actor.is_scim_idp and silo_group_list.silo in actor.silo;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest)\n if actor.is_user;\nhas_permission(actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest)\n if actor.is_user;\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_alert_receiver: AlertReceiver };\n\n\t"read" if "read" on "parent_alert_receiver";\n\t"modify" if "modify" on "parent_alert_receiver";\n}\n\nhas_relation(rx: AlertReceiver, "parent_alert_receiver", secret: WebhookSecret)\n\tif secret.alert_receiver = rx;\n\nresource AlertClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: AlertClassList)\n\tif collection.fleet = fleet;\n\nresource ScimClientBearerTokenList {\n\tpermissions = [ "create_child", "list_children" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges for SCIM client tokens.\n # These are all admin because being able to create these tokens would allow\n # a user to grant themselves admin by modifying group membership through SCIM calls\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_silo";\n\n # Fleet-level roles also grant privileges for SCIM client tokens, for\n # configuration before silo admins are present.\n\t"create_child" if "admin" on "parent_fleet";\n\t"list_children" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", scim_client_bearer_token_list: ScimClientBearerTokenList)\n\tif scim_client_bearer_token_list.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: ScimClientBearerTokenList)\n\tif collection.silo.fleet = fleet;\n\n# VpcList is a synthetic resource for controlling VPC creation.\n# Unlike other project resources, VPC creation requires the full "collaborator"\n# role rather than "limited-collaborator", enforcing the networking restriction.\n# This allows organizations to restrict who can reconfigure the network topology\n# while still allowing users with limited-collaborator to work with compute\n# resources (instances, disks, etc.) within the existing network.\nresource VpcList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { containing_project: Project };\n\n\t"list_children" if "read" on "containing_project";\n\t"create_child" if "collaborator" on "containing_project";\n}\nhas_relation(project: Project, "containing_project", collection: VpcList)\n\tif collection.project = project;\n\n# Describes the policy for accessing "/v1/system/subnet-pools" in the API\nresource SubnetPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the Subnet Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list External Subnet Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", subnet_pool_list: SubnetPoolList)\n\tif subnet_pool_list.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource ExternalSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ExternalSubnet)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource SubnetPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SubnetPool)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource TufTrustRoot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufTrustRoot)\n if child.fleet = fleet;\n \n\n resource Alert {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Alert)\n if child.fleet = fleet;\n \n\n resource AlertReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AlertReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n \n\n
1122026-03-10T20:08:03.114ZINFOtest_retry_backoff (ServerContext): Setting up resolver using DNS address: [::1]:49304
1132026-03-10T20:08:03.115ZINFOtest_retry_backoff (DnsResolver): new DNS resolver addresses = [[::1]:49304]
1142026-03-10T20:08:03.115ZINFOtest_retry_backoff (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:42738/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 42738,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1152026-03-10T20:08:03.117ZDEBGtest_retry_backoff (ServerContext): registered USDT probes
1162026-03-10T20:08:04.161ZINFOtest_retry_backoff (datastore): Database schema version is up to date desired_version = 237.0.0 found_version = 237.0.0
1172026-03-10T20:08:04.177ZWARNtest_retry_backoff (datastore): No db_metadata_nexus records exist - skipping access check explanation = This is expected during initial deployment or before migration nexus_id = 913233fe-92a8-4635-9572-183f495429c4 (omicron_zone)
1182026-03-10T20:08:04.177ZINFOtest_retry_backoff (nexus): Datastore is ready for usage
1192026-03-10T20:08:04.307ZDEBGtest_retry_backoff (clickhouse-client): registered USDT probes id = 1f627a34-b3eb-405b-a511-ad3e20ee2606
1202026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1212026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1222026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1232026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1242026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1252026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1262026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = PUT path = /disks/{disk_id} versions = all
1272026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /metrics/collectors versions = all
1282026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1292026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /metrics/producers versions = all
1302026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1312026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = GET path = /probes/{sled} versions = all
1322026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /refresh-vpc-routes versions = all
1332026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = GET path = /sled-agents/{sled_id} versions = all
1342026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /sled-agents/{sled_id} versions = all
1352026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = PUT path = /switch/{switch_id} versions = all
1362026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = GET path = /v1/ping versions = all
1372026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = PUT path = /vmms/{propolis_id} versions = all
1382026-03-10T20:08:04.360ZDEBGtest_retry_backoff (dropshot_internal): registered endpoint local_addr = 127.0.0.1:63060 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1392026-03-10T20:08:04.360ZINFOtest_retry_backoff (dropshot_internal): listening local_addr = 127.0.0.1:63060
1402026-03-10T20:08:04.361ZDEBGtest_retry_backoff (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:63060
1412026-03-10T20:08:04.363ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /bgtasks versions = all
1422026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /bgtasks/activate versions = all
1432026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1442026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /clickhouse/policy versions = all
1452026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /clickhouse/policy versions = all
1462026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /debug/fetch-omdb-binary versions = all
1472026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /demo-saga versions = all
1482026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1492026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /deployment/blueprints/all versions = all
1502026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1512026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1522026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /deployment/blueprints/import versions = all
1532026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /deployment/blueprints/regenerate versions = all
1542026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /deployment/blueprints/target versions = all
1552026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /deployment/blueprints/target versions = all
1562026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = PUT path = /deployment/blueprints/target/enabled versions = all
1572026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /deployment/reconfigurator-config versions = all
1582026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /deployment/reconfigurator-config versions = all
1592026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /deployment/reconfigurator-config/{version} versions = all
1602026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /deployment/update-status versions = all
1612026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /experimental/v1/system/support-bundles versions = all
1622026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /experimental/v1/system/support-bundles versions = all
1632026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = DELETE path = /experimental/v1/system/support-bundles/{bundle_id} versions = all
1642026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /experimental/v1/system/support-bundles/{bundle_id} versions = all
1652026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = PUT path = /experimental/v1/system/support-bundles/{bundle_id} versions = all
1662026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /experimental/v1/system/support-bundles/{bundle_id}/download versions = all
1672026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = HEAD path = /experimental/v1/system/support-bundles/{bundle_id}/download versions = all
1682026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /experimental/v1/system/support-bundles/{bundle_id}/download/{file} versions = all
1692026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = HEAD path = /experimental/v1/system/support-bundles/{bundle_id}/download/{file} versions = all
1702026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /experimental/v1/system/support-bundles/{bundle_id}/index versions = all
1712026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /instances/{instance_id}/migrate versions = all
1722026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /mgs-updates versions = all
1732026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /oximeter/read-policy versions = all
1742026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /oximeter/read-policy versions = all
1752026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /physical-disk/expunge versions = all
1762026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /quiesce versions = all
1772026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /quiesce versions = all
1782026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1792026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /sagas versions = all
1802026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /sagas/{saga_id} versions = all
1812026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /sleds/expunge versions = all
1822026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /sleds/uninitialized versions = all
1832026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /trust-quorum/config/{rack_id} versions = all
1842026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /trust-quorum/lrtq-upgrade versions = all
1852026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = POST path = /trust-quorum/remove/{sled} versions = all
1862026-03-10T20:08:04.364ZDEBGtest_retry_backoff (dropshot_lockstep): registered endpoint local_addr = 127.0.0.1:50070 method = GET path = /v1/ping versions = all
1872026-03-10T20:08:04.365ZINFOtest_retry_backoff (dropshot_lockstep): listening local_addr = 127.0.0.1:50070
1882026-03-10T20:08:04.365ZDEBGtest_retry_backoff (dropshot_lockstep): successfully registered DTrace USDT probes local_addr = 127.0.0.1:50070
1892026-03-10T20:08:04.365ZINFOtest_retry_backoff (SEC): SEC running sec_id = 913233fe-92a8-4635-9572-183f495429c4
1902026-03-10T20:08:04.366ZTRACtest_retry_backoff (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
1912026-03-10T20:08:04.366ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
1922026-03-10T20:08:04.368ZDEBGtest_retry_backoff (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
1932026-03-10T20:08:04.368ZDEBGtest_retry_backoff (DataLoader): attempting to create built-in users
1942026-03-10T20:08:04.369ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
1952026-03-10T20:08:04.369ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
1962026-03-10T20:08:04.370ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
1972026-03-10T20:08:04.540ZINFOtest_retry_backoff (MgsUpdateDriver): starting MgsUpdateDriver
1982026-03-10T20:08:04.543ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d2 84 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56832 req_id = 6f30b233-8d74-4318-b40b-e6d552b3470f
1992026-03-10T20:08:04.544ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53892,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:56832 req_id = 6f30b233-8d74-4318-b40b-e6d552b3470f
2002026-03-10T20:08:04.545ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53892, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56832 req_id = 6f30b233-8d74-4318-b40b-e6d552b3470f
2012026-03-10T20:08:04.545ZTRACtest_retry_backoff (dns): buffer buffer = Length: 74 (0x4a) bytes\n0000: 90 6c 01 00 00 01 00 00 00 00 00 01 0b 5f 72 65 .l..........._re\n0010: 70 6f 2d 64 65 70 6f 74 04 5f 74 63 70 0d 63 6f po-depot._tcp.co\n0020: 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 ntrol-plane.oxid\n0030: 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 e.internal..!...\n0040: 00 29 04 d0 00 00 00 00 00 00 .)........ kind = dns peer_addr = [::1]:50883 req_id = c1715239-c7c1-4284-b603-17f93d772f69
2022026-03-10T20:08:04.545ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36972,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_repo-depot._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_repo-depot._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 11,\n 95,\n 114,\n 101,\n 112,\n 111,\n 45,\n 100,\n 101,\n 112,\n 111,\n 116,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:50883 req_id = c1715239-c7c1-4284-b603-17f93d772f69
2032026-03-10T20:08:04.545ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36972, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_repo-depot._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_repo-depot._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 11, 95, 114, 101, 112, 111, 45, 100, 101, 112, 111, 116, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_repo-depot._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50883 req_id = c1715239-c7c1-4284-b603-17f93d772f69
2042026-03-10T20:08:04.546ZTRACtest_retry_backoff (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 20 7b 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 {..........._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:61437 req_id = 1016b91f-7b83-4487-b160-a3a109613502
2052026-03-10T20:08:04.546ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8315,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:61437 req_id = 1016b91f-7b83-4487-b160-a3a109613502
2062026-03-10T20:08:04.546ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8315, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61437 req_id = 1016b91f-7b83-4487-b160-a3a109613502
2072026-03-10T20:08:04.549ZINFOtest_retry_backoff (DataLoader): created 5 built-in users
2082026-03-10T20:08:04.549ZTRACtest_retry_backoff (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2092026-03-10T20:08:04.549ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2102026-03-10T20:08:04.551ZDEBGtest_retry_backoff (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2112026-03-10T20:08:04.551ZDEBGtest_retry_backoff (DataLoader): attempting to create built-in role assignments
2122026-03-10T20:08:04.551ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2132026-03-10T20:08:04.551ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2142026-03-10T20:08:04.551ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2152026-03-10T20:08:04.671ZINFOtest_retry_backoff (DataLoader): created 4 built-in role assignments
2162026-03-10T20:08:04.671ZTRACtest_retry_backoff (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2172026-03-10T20:08:04.671ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2182026-03-10T20:08:04.673ZDEBGtest_retry_backoff (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2192026-03-10T20:08:04.673ZDEBGtest_retry_backoff (DataLoader): attempting to create built-in silos
2202026-03-10T20:08:04.673ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2212026-03-10T20:08:04.673ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2222026-03-10T20:08:04.674ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2232026-03-10T20:08:04.740ZINFOtest_retry_backoff (DataLoader): created 2 built-in silos
2242026-03-10T20:08:04.740ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2252026-03-10T20:08:04.740ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2262026-03-10T20:08:04.741ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2272026-03-10T20:08:04.749ZTRACtest_retry_backoff (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2282026-03-10T20:08:04.749ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2292026-03-10T20:08:04.751ZDEBGtest_retry_backoff (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2302026-03-10T20:08:04.751ZDEBGtest_retry_backoff (DataLoader): attempting to create built-in projects
2312026-03-10T20:08:04.751ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2322026-03-10T20:08:04.751ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2332026-03-10T20:08:04.752ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2342026-03-10T20:08:04.759ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2352026-03-10T20:08:04.759ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2362026-03-10T20:08:04.759ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2372026-03-10T20:08:04.759ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2382026-03-10T20:08:04.760ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2392026-03-10T20:08:04.803ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2402026-03-10T20:08:04.803ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2412026-03-10T20:08:04.803ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2422026-03-10T20:08:04.804ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2432026-03-10T20:08:04.806ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2442026-03-10T20:08:04.809ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2452026-03-10T20:08:04.810ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2462026-03-10T20:08:04.810ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2472026-03-10T20:08:04.810ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2482026-03-10T20:08:04.810ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2492026-03-10T20:08:04.810ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2502026-03-10T20:08:04.812ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2512026-03-10T20:08:04.812ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2522026-03-10T20:08:04.812ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2532026-03-10T20:08:04.812ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2542026-03-10T20:08:04.814ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2552026-03-10T20:08:04.817ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2562026-03-10T20:08:04.817ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2572026-03-10T20:08:04.817ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2582026-03-10T20:08:04.817ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2592026-03-10T20:08:04.829ZINFOtest_retry_backoff (DataLoader): created built-in services project
2602026-03-10T20:08:04.829ZTRACtest_retry_backoff (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2612026-03-10T20:08:04.829ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2622026-03-10T20:08:04.831ZDEBGtest_retry_backoff (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2632026-03-10T20:08:04.831ZDEBGtest_retry_backoff (DataLoader): attempting to create built-in VPCs
2642026-03-10T20:08:04.831ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2652026-03-10T20:08:04.831ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2662026-03-10T20:08:04.831ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2672026-03-10T20:08:04.834ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2682026-03-10T20:08:04.834ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2692026-03-10T20:08:04.834ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2702026-03-10T20:08:04.836ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2712026-03-10T20:08:04.836ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2722026-03-10T20:08:04.836ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2732026-03-10T20:08:04.836ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2742026-03-10T20:08:04.836ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2752026-03-10T20:08:04.838ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2762026-03-10T20:08:04.838ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2772026-03-10T20:08:04.838ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2782026-03-10T20:08:04.838ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2792026-03-10T20:08:04.839ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2802026-03-10T20:08:04.840ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2812026-03-10T20:08:04.840ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2822026-03-10T20:08:04.840ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2832026-03-10T20:08:04.842ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2842026-03-10T20:08:04.846ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2852026-03-10T20:08:04.846ZTRACtest_retry_backoff (DataLoader): successful lookup of siloed resource "Project" using built-in user
2862026-03-10T20:08:04.846ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2872026-03-10T20:08:04.846ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2882026-03-10T20:08:04.846ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2892026-03-10T20:08:04.846ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2902026-03-10T20:08:04.847ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2912026-03-10T20:08:04.854ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2922026-03-10T20:08:04.854ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2932026-03-10T20:08:04.854ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2942026-03-10T20:08:04.855ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2952026-03-10T20:08:04.857ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2962026-03-10T20:08:04.857ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
2972026-03-10T20:08:04.857ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
2982026-03-10T20:08:04.857ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
2992026-03-10T20:08:04.859ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3002026-03-10T20:08:04.864ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
3012026-03-10T20:08:04.864ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3022026-03-10T20:08:04.864ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3032026-03-10T20:08:04.864ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3042026-03-10T20:08:04.874ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3052026-03-10T20:08:04.874ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3062026-03-10T20:08:04.874ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3072026-03-10T20:08:04.874ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3082026-03-10T20:08:04.874ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3092026-03-10T20:08:04.877ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3102026-03-10T20:08:04.877ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3112026-03-10T20:08:04.877ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3122026-03-10T20:08:04.877ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3132026-03-10T20:08:04.879ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3142026-03-10T20:08:04.879ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3152026-03-10T20:08:04.879ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3162026-03-10T20:08:04.879ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3172026-03-10T20:08:04.881ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3182026-03-10T20:08:04.885ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3192026-03-10T20:08:04.885ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3202026-03-10T20:08:04.885ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3212026-03-10T20:08:04.885ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3222026-03-10T20:08:04.890ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3232026-03-10T20:08:04.890ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3242026-03-10T20:08:04.890ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3252026-03-10T20:08:04.895ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3262026-03-10T20:08:04.895ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3272026-03-10T20:08:04.895ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3282026-03-10T20:08:04.898ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3292026-03-10T20:08:04.898ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3302026-03-10T20:08:04.899ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3312026-03-10T20:08:04.899ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3322026-03-10T20:08:04.899ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3332026-03-10T20:08:04.905ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3342026-03-10T20:08:04.905ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3352026-03-10T20:08:04.905ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3362026-03-10T20:08:04.905ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3372026-03-10T20:08:04.907ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3382026-03-10T20:08:04.907ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3392026-03-10T20:08:04.907ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3402026-03-10T20:08:04.907ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3412026-03-10T20:08:04.909ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3422026-03-10T20:08:04.913ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3432026-03-10T20:08:04.913ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3442026-03-10T20:08:04.913ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3452026-03-10T20:08:04.914ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3462026-03-10T20:08:04.957ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3472026-03-10T20:08:04.957ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3482026-03-10T20:08:04.957ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3492026-03-10T20:08:04.957ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3502026-03-10T20:08:04.957ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3512026-03-10T20:08:04.959ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3522026-03-10T20:08:04.959ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3532026-03-10T20:08:04.959ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3542026-03-10T20:08:04.960ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3552026-03-10T20:08:04.962ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3562026-03-10T20:08:04.962ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3572026-03-10T20:08:04.962ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3582026-03-10T20:08:04.962ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3592026-03-10T20:08:04.963ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3602026-03-10T20:08:04.968ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3612026-03-10T20:08:04.968ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3622026-03-10T20:08:04.968ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3632026-03-10T20:08:04.969ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3642026-03-10T20:08:05.028ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3652026-03-10T20:08:05.028ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3662026-03-10T20:08:05.028ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3672026-03-10T20:08:05.028ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3682026-03-10T20:08:05.028ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3692026-03-10T20:08:05.032ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3702026-03-10T20:08:05.032ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3712026-03-10T20:08:05.032ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3722026-03-10T20:08:05.032ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3732026-03-10T20:08:05.034ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3742026-03-10T20:08:05.034ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3752026-03-10T20:08:05.034ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3762026-03-10T20:08:05.035ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3772026-03-10T20:08:05.036ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3782026-03-10T20:08:05.041ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3792026-03-10T20:08:05.041ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3802026-03-10T20:08:05.041ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3812026-03-10T20:08:05.041ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3822026-03-10T20:08:05.045ZDEBGtest_retry_backoff (DataLoader): attempting to create built-in VPC firewall rules
3832026-03-10T20:08:05.045ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3842026-03-10T20:08:05.045ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3852026-03-10T20:08:05.046ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3862026-03-10T20:08:05.049ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3872026-03-10T20:08:05.049ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3882026-03-10T20:08:05.049ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3892026-03-10T20:08:05.051ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3902026-03-10T20:08:05.051ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3912026-03-10T20:08:05.051ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3922026-03-10T20:08:05.053ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3932026-03-10T20:08:05.053ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3942026-03-10T20:08:05.053ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3952026-03-10T20:08:05.053ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
3962026-03-10T20:08:05.053ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
3972026-03-10T20:08:05.055ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3982026-03-10T20:08:05.055ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
3992026-03-10T20:08:05.055ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4002026-03-10T20:08:05.055ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4012026-03-10T20:08:05.056ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4022026-03-10T20:08:05.056ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4032026-03-10T20:08:05.056ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4042026-03-10T20:08:05.057ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4052026-03-10T20:08:05.058ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4062026-03-10T20:08:05.062ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4072026-03-10T20:08:05.062ZTRACtest_retry_backoff (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4082026-03-10T20:08:05.062ZTRACtest_retry_backoff (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4092026-03-10T20:08:05.062ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4102026-03-10T20:08:05.062ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4112026-03-10T20:08:05.062ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4122026-03-10T20:08:05.063ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4132026-03-10T20:08:05.064ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4142026-03-10T20:08:05.064ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4152026-03-10T20:08:05.064ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4162026-03-10T20:08:05.065ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4172026-03-10T20:08:05.066ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4182026-03-10T20:08:05.066ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4192026-03-10T20:08:05.066ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4202026-03-10T20:08:05.066ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4212026-03-10T20:08:05.067ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4222026-03-10T20:08:05.072ZDEBGtest_retry_backoff (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4232026-03-10T20:08:05.073ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4242026-03-10T20:08:05.073ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4252026-03-10T20:08:05.073ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4262026-03-10T20:08:05.079ZTRACtest_retry_backoff (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4272026-03-10T20:08:05.079ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4282026-03-10T20:08:05.079ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4292026-03-10T20:08:05.079ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4302026-03-10T20:08:05.080ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4312026-03-10T20:08:05.082ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4322026-03-10T20:08:05.082ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4332026-03-10T20:08:05.082ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4342026-03-10T20:08:05.082ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4352026-03-10T20:08:05.084ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4362026-03-10T20:08:05.084ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4372026-03-10T20:08:05.084ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4382026-03-10T20:08:05.084ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4392026-03-10T20:08:05.085ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4402026-03-10T20:08:05.089ZDEBGtest_retry_backoff (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4412026-03-10T20:08:05.089ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4422026-03-10T20:08:05.089ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4432026-03-10T20:08:05.090ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4442026-03-10T20:08:05.146ZINFOtest_retry_backoff (DataLoader): created built-in VPC firewall rules
4452026-03-10T20:08:05.146ZDEBGtest_retry_backoff (DataLoader): attempting to create built-in VPC Subnets
4462026-03-10T20:08:05.146ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4472026-03-10T20:08:05.146ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4482026-03-10T20:08:05.147ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4492026-03-10T20:08:05.151ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4502026-03-10T20:08:05.151ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4512026-03-10T20:08:05.151ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4522026-03-10T20:08:05.153ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4532026-03-10T20:08:05.153ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4542026-03-10T20:08:05.154ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4552026-03-10T20:08:05.155ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4562026-03-10T20:08:05.156ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4572026-03-10T20:08:05.156ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4582026-03-10T20:08:05.156ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4592026-03-10T20:08:05.156ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4602026-03-10T20:08:05.158ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4612026-03-10T20:08:05.158ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4622026-03-10T20:08:05.158ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4632026-03-10T20:08:05.158ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4642026-03-10T20:08:05.160ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4652026-03-10T20:08:05.160ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4662026-03-10T20:08:05.160ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4672026-03-10T20:08:05.160ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4682026-03-10T20:08:05.162ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4692026-03-10T20:08:05.166ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4702026-03-10T20:08:05.166ZTRACtest_retry_backoff (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4712026-03-10T20:08:05.166ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4722026-03-10T20:08:05.166ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4732026-03-10T20:08:05.167ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4742026-03-10T20:08:05.172ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4752026-03-10T20:08:05.172ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4762026-03-10T20:08:05.172ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4772026-03-10T20:08:05.172ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4782026-03-10T20:08:05.172ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4792026-03-10T20:08:05.174ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4802026-03-10T20:08:05.174ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4812026-03-10T20:08:05.174ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4822026-03-10T20:08:05.174ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4832026-03-10T20:08:05.176ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4842026-03-10T20:08:05.176ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4852026-03-10T20:08:05.176ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4862026-03-10T20:08:05.176ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4872026-03-10T20:08:05.178ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4882026-03-10T20:08:05.187ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4892026-03-10T20:08:05.192ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4902026-03-10T20:08:05.192ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4912026-03-10T20:08:05.192ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4922026-03-10T20:08:05.192ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4932026-03-10T20:08:05.192ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4942026-03-10T20:08:05.194ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4952026-03-10T20:08:05.194ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
4962026-03-10T20:08:05.194ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
4972026-03-10T20:08:05.194ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
4982026-03-10T20:08:05.198ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4992026-03-10T20:08:05.198ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5002026-03-10T20:08:05.198ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5012026-03-10T20:08:05.198ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5022026-03-10T20:08:05.200ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5032026-03-10T20:08:05.211ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5042026-03-10T20:08:05.211ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5052026-03-10T20:08:05.211ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5062026-03-10T20:08:05.211ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5072026-03-10T20:08:05.255ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5082026-03-10T20:08:05.255ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5092026-03-10T20:08:05.256ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5102026-03-10T20:08:05.258ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5112026-03-10T20:08:05.258ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5122026-03-10T20:08:05.258ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5132026-03-10T20:08:05.258ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5142026-03-10T20:08:05.258ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5152026-03-10T20:08:05.259ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5162026-03-10T20:08:05.259ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5172026-03-10T20:08:05.259ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5182026-03-10T20:08:05.260ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5192026-03-10T20:08:05.261ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5202026-03-10T20:08:05.261ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5212026-03-10T20:08:05.261ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5222026-03-10T20:08:05.261ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5232026-03-10T20:08:05.262ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5242026-03-10T20:08:05.267ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5252026-03-10T20:08:05.269ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5262026-03-10T20:08:05.269ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5272026-03-10T20:08:05.269ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5282026-03-10T20:08:05.269ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5292026-03-10T20:08:05.270ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5302026-03-10T20:08:05.271ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5312026-03-10T20:08:05.271ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5322026-03-10T20:08:05.271ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5332026-03-10T20:08:05.271ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5342026-03-10T20:08:05.273ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5352026-03-10T20:08:05.273ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5362026-03-10T20:08:05.273ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5372026-03-10T20:08:05.273ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5382026-03-10T20:08:05.274ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5392026-03-10T20:08:05.279ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5402026-03-10T20:08:05.279ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5412026-03-10T20:08:05.279ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5422026-03-10T20:08:05.279ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5432026-03-10T20:08:05.283ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5442026-03-10T20:08:05.283ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5452026-03-10T20:08:05.284ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5462026-03-10T20:08:05.285ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5472026-03-10T20:08:05.286ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5482026-03-10T20:08:05.286ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5492026-03-10T20:08:05.286ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5502026-03-10T20:08:05.286ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5512026-03-10T20:08:05.287ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5522026-03-10T20:08:05.287ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5532026-03-10T20:08:05.287ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5542026-03-10T20:08:05.288ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5552026-03-10T20:08:05.289ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5562026-03-10T20:08:05.289ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5572026-03-10T20:08:05.289ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5582026-03-10T20:08:05.289ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5592026-03-10T20:08:05.291ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5602026-03-10T20:08:05.295ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5612026-03-10T20:08:05.298ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5622026-03-10T20:08:05.298ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5632026-03-10T20:08:05.298ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5642026-03-10T20:08:05.298ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5652026-03-10T20:08:05.298ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5662026-03-10T20:08:05.299ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5672026-03-10T20:08:05.300ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5682026-03-10T20:08:05.300ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5692026-03-10T20:08:05.300ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5702026-03-10T20:08:05.301ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5712026-03-10T20:08:05.301ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5722026-03-10T20:08:05.301ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5732026-03-10T20:08:05.302ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5742026-03-10T20:08:05.303ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5752026-03-10T20:08:05.311ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5762026-03-10T20:08:05.311ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5772026-03-10T20:08:05.311ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5782026-03-10T20:08:05.312ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5792026-03-10T20:08:05.317ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5802026-03-10T20:08:05.317ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5812026-03-10T20:08:05.317ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5822026-03-10T20:08:05.319ZINFOtest_retry_backoff (DataLoader): created built-in services vpc subnets
5832026-03-10T20:08:05.319ZINFOtest_retry_backoff (DataLoader): created built-in services vpc
5842026-03-10T20:08:05.319ZTRACtest_retry_backoff (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5852026-03-10T20:08:05.319ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5862026-03-10T20:08:05.321ZDEBGtest_retry_backoff (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5872026-03-10T20:08:05.321ZDEBGtest_retry_backoff (DataLoader): attempting to create silo users
5882026-03-10T20:08:05.321ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5892026-03-10T20:08:05.321ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5902026-03-10T20:08:05.321ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5912026-03-10T20:08:05.326ZINFOtest_retry_backoff (DataLoader): created 2 silo users
5922026-03-10T20:08:05.326ZDEBGtest_retry_backoff (DataLoader): attempting to create silo user role assignments
5932026-03-10T20:08:05.326ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5942026-03-10T20:08:05.326ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5952026-03-10T20:08:05.327ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
5962026-03-10T20:08:05.329ZINFOtest_retry_backoff (DataLoader): created 2 silo user role assignments
5972026-03-10T20:08:05.329ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
5982026-03-10T20:08:05.329ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
5992026-03-10T20:08:05.329ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
6002026-03-10T20:08:05.332ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
6012026-03-10T20:08:05.332ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
6022026-03-10T20:08:05.332ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
6032026-03-10T20:08:05.336ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = IpPoolList
6042026-03-10T20:08:05.337ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6052026-03-10T20:08:05.337ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
6062026-03-10T20:08:05.337ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
6072026-03-10T20:08:05.337ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
6082026-03-10T20:08:05.338ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
6092026-03-10T20:08:05.340ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = IpPoolList result = Ok(())
6102026-03-10T20:08:05.340ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
6112026-03-10T20:08:05.340ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
6122026-03-10T20:08:05.340ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
6132026-03-10T20:08:05.452ZTRACtest_retry_backoff (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = IpPoolList
6142026-03-10T20:08:05.452ZTRACtest_retry_backoff (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6152026-03-10T20:08:05.452ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
6162026-03-10T20:08:05.452ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
6172026-03-10T20:08:05.453ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
6182026-03-10T20:08:05.455ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
6192026-03-10T20:08:05.456ZDEBGtest_retry_backoff (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = IpPoolList result = Ok(())
6202026-03-10T20:08:05.456ZTRACtest_retry_backoff (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database
6212026-03-10T20:08:05.456ZDEBGtest_retry_backoff (DataLoader): roles roles = RoleSet { roles: {} }
6222026-03-10T20:08:05.456ZDEBGtest_retry_backoff (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001 (built_in_user), .. }) resource = Database result = Ok(())
6232026-03-10T20:08:05.458ZINFOtest_retry_backoff (nexus): populate complete
6242026-03-10T20:08:05.459ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = BlueprintConfig
6252026-03-10T20:08:05.459ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6262026-03-10T20:08:05.459ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
6272026-03-10T20:08:05.459ZDEBGtest_retry_backoff (BackgroundTasks): roles roles = RoleSet { roles: {} }
6282026-03-10T20:08:05.459ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
6292026-03-10T20:08:05.506ZDEBGtest_retry_backoff (init_with_steps): Running step start_gateway_switch0
6302026-03-10T20:08:05.507ZDEBGtest_retry_backoff: Starting Management Gateway
6312026-03-10T20:08:05.535ZDEBGtest_retry_backoff (BackgroundTasks): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6322026-03-10T20:08:05.538ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = BlueprintConfig result = Ok(())
6332026-03-10T20:08:05.539ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
6342026-03-10T20:08:05.539ZDEBGtest_retry_backoff (BackgroundTasks): roles roles = RoleSet { roles: {} }
6352026-03-10T20:08:05.539ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
6362026-03-10T20:08:05.544ZINFOtest_retry_backoff (nexus): activating background tasks
6372026-03-10T20:08:05.548ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6382026-03-10T20:08:05.548ZDEBGtest_retry_backoff (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6392026-03-10T20:08:05.548ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database
6402026-03-10T20:08:05.548ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6412026-03-10T20:08:05.549ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6422026-03-10T20:08:05.551ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6432026-03-10T20:08:05.551ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6442026-03-10T20:08:05.552ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6452026-03-10T20:08:05.552ZWARNtest_retry_backoff (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6462026-03-10T20:08:05.552ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 132.514\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6472026-03-10T20:08:05.552ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6482026-03-10T20:08:05.552ZDEBGtest_retry_backoff (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6492026-03-10T20:08:05.552ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database
6502026-03-10T20:08:05.552ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6512026-03-10T20:08:05.552ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6522026-03-10T20:08:05.553ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
6532026-03-10T20:08:05.553ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
6542026-03-10T20:08:05.553ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
6552026-03-10T20:08:05.553ZWARNtest_retry_backoff (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
6562026-03-10T20:08:05.553ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 20.945\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6572026-03-10T20:08:05.553ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
6582026-03-10T20:08:05.553ZINFOtest_retry_backoff (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2026-03-10 19:58:05.553338644 UTC
6592026-03-10T20:08:05.553ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = metrics_producer_gc resource = Database
6602026-03-10T20:08:05.553ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
6612026-03-10T20:08:05.553ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
6622026-03-10T20:08:05.554ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
6632026-03-10T20:08:05.554ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Fleet
6642026-03-10T20:08:05.554ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6652026-03-10T20:08:05.554ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database
6662026-03-10T20:08:05.554ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6672026-03-10T20:08:05.554ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database result = Ok(())
6682026-03-10T20:08:05.557ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = nat_garbage_collector iteration = 1 reason = Timeout
6692026-03-10T20:08:05.557ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = nat_garbage_collector resource = Database
6702026-03-10T20:08:05.557ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = nat_garbage_collector roles = RoleSet { roles: {} }
6712026-03-10T20:08:05.558ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = nat_garbage_collector resource = Database result = Ok(())
6722026-03-10T20:08:05.558ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
6732026-03-10T20:08:05.558ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = bfd_manager resource = Database
6742026-03-10T20:08:05.558ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6752026-03-10T20:08:05.558ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = bfd_manager resource = Database result = Ok(())
6762026-03-10T20:08:05.559ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
6772026-03-10T20:08:05.559ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
6782026-03-10T20:08:05.559ZWARNtest_retry_backoff (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
6792026-03-10T20:08:05.559ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 36.449\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6802026-03-10T20:08:05.559ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
6812026-03-10T20:08:05.559ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Inventory
6822026-03-10T20:08:05.559ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6832026-03-10T20:08:05.559ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
6842026-03-10T20:08:05.559ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
6852026-03-10T20:08:05.559ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
6862026-03-10T20:08:05.560ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = inventory_loader iteration = 1 reason = Timeout
6872026-03-10T20:08:05.560ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Inventory
6882026-03-10T20:08:05.560ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = inventory_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6892026-03-10T20:08:05.560ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database
6902026-03-10T20:08:05.560ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {} }
6912026-03-10T20:08:05.560ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database result = Ok(())
6922026-03-10T20:08:05.561ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = reconfigurator_config_watcher iteration = 1 reason = Timeout
6932026-03-10T20:08:05.561ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig
6942026-03-10T20:08:05.561ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = reconfigurator_config_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6952026-03-10T20:08:05.561ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
6962026-03-10T20:08:05.561ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
6972026-03-10T20:08:05.561ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
6982026-03-10T20:08:05.561ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = blueprint_planner iteration = 1 reason = Timeout
6992026-03-10T20:08:05.561ZDEBGtest_retry_backoff (BackgroundTasks): reconfigurator config not yet loaded; doing nothing background_task = blueprint_planner
7002026-03-10T20:08:05.562ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 75.64\u{b5}s iteration = 1 status = "Disabled"
7012026-03-10T20:08:05.562ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
7022026-03-10T20:08:05.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = BlueprintConfig
7032026-03-10T20:08:05.562ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7042026-03-10T20:08:05.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database
7052026-03-10T20:08:05.562ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7062026-03-10T20:08:05.562ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database result = Ok(())
7072026-03-10T20:08:05.562ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
7082026-03-10T20:08:05.562ZWARNtest_retry_backoff (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
7092026-03-10T20:08:05.562ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 22.806\u{b5}s iteration = 1 status = {"error":"no blueprint"}
7102026-03-10T20:08:05.562ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
7112026-03-10T20:08:05.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Fleet
7122026-03-10T20:08:05.562ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7132026-03-10T20:08:05.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Database
7142026-03-10T20:08:05.562ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7152026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7162026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
7172026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 36.88\u{b5}s iteration = 1 status = {"error":"task disabled"}
7182026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
7192026-03-10T20:08:05.563ZWARNtest_retry_backoff (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
7202026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 8.532\u{b5}s iteration = 1 status = {"error":"no blueprint"}
7212026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
7222026-03-10T20:08:05.563ZINFOtest_retry_backoff (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
7232026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 83.291\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
7242026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
7252026-03-10T20:08:05.563ZWARNtest_retry_backoff (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
7262026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 24.536\u{b5}s iteration = 1 status = {"error":"inventory collection is None"}
7272026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
7282026-03-10T20:08:05.563ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Fleet
7292026-03-10T20:08:05.563ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7302026-03-10T20:08:05.563ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database
7312026-03-10T20:08:05.563ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7322026-03-10T20:08:05.564ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7332026-03-10T20:08:05.564ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
7342026-03-10T20:08:05.565ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database
7352026-03-10T20:08:05.565ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7362026-03-10T20:08:05.565ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database result = Ok(())
7372026-03-10T20:08:05.565ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
7382026-03-10T20:08:05.565ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_replacement resource = Database
7392026-03-10T20:08:05.565ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7402026-03-10T20:08:05.566ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_replacement resource = Database result = Ok(())
7412026-03-10T20:08:05.566ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7422026-03-10T20:08:05.566ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_replacement_driver resource = Database
7432026-03-10T20:08:05.566ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7442026-03-10T20:08:05.566ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7452026-03-10T20:08:05.566ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7462026-03-10T20:08:05.567ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Fleet
7472026-03-10T20:08:05.567ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7482026-03-10T20:08:05.567ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Database
7492026-03-10T20:08:05.567ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7502026-03-10T20:08:05.567ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Database result = Ok(())
7512026-03-10T20:08:05.567ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7522026-03-10T20:08:05.567ZINFOtest_retry_backoff (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7532026-03-10T20:08:05.567ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = instance_updater elapsed = 275.121\u{b5}s iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7542026-03-10T20:08:05.568ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7552026-03-10T20:08:05.568ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_reincarnation resource = Database
7562026-03-10T20:08:05.568ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7572026-03-10T20:08:05.568ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7582026-03-10T20:08:05.568ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7592026-03-10T20:08:05.568ZDEBGtest_retry_backoff (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7602026-03-10T20:08:05.568ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
7612026-03-10T20:08:05.568ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7622026-03-10T20:08:05.569ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7632026-03-10T20:08:05.569ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
7642026-03-10T20:08:05.569ZINFOtest_retry_backoff (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
7652026-03-10T20:08:05.569ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet
7662026-03-10T20:08:05.569ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7672026-03-10T20:08:05.569ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
7682026-03-10T20:08:05.569ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7692026-03-10T20:08:05.569ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7702026-03-10T20:08:05.569ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
7712026-03-10T20:08:05.570ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = abandoned_vmm_reaper resource = Database
7722026-03-10T20:08:05.570ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
7732026-03-10T20:08:05.570ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
7742026-03-10T20:08:05.570ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
7752026-03-10T20:08:05.570ZINFOtest_retry_backoff (SagaQuiesceHandle): saga recovery pass starting
7762026-03-10T20:08:05.570ZDEBGtest_retry_backoff (SagaRecoverer): listing candidate sagas for recovery
7772026-03-10T20:08:05.570ZTRACtest_retry_backoff (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a (built_in_user), .. }) resource = Database
7782026-03-10T20:08:05.570ZDEBGtest_retry_backoff (SagaRecoverer): roles roles = RoleSet { roles: {} }
7792026-03-10T20:08:05.571ZDEBGtest_retry_backoff (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a (built_in_user), .. }) resource = Database result = Ok(())
7802026-03-10T20:08:05.598ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
7812026-03-10T20:08:05.598ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = lookup_region_port resource = Database
7822026-03-10T20:08:05.598ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
7832026-03-10T20:08:05.598ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = lookup_region_port resource = Database result = Ok(())
7842026-03-10T20:08:05.599ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
7852026-03-10T20:08:05.599ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_start resource = Database
7862026-03-10T20:08:05.599ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7872026-03-10T20:08:05.599ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
7882026-03-10T20:08:05.599ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
7892026-03-10T20:08:05.599ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
7902026-03-10T20:08:05.599ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
7912026-03-10T20:08:05.600ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
7922026-03-10T20:08:05.600ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
7932026-03-10T20:08:05.600ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_step resource = Database
7942026-03-10T20:08:05.600ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7952026-03-10T20:08:05.600ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7962026-03-10T20:08:05.601ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
7972026-03-10T20:08:05.601ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_finish resource = Database
7982026-03-10T20:08:05.601ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
7992026-03-10T20:08:05.601ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
8002026-03-10T20:08:05.601ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
8012026-03-10T20:08:05.601ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Fleet
8022026-03-10T20:08:05.601ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8032026-03-10T20:08:05.601ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database
8042026-03-10T20:08:05.601ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
8052026-03-10T20:08:05.601ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
8062026-03-10T20:08:05.602ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = tuf_repo_pruner iteration = 1 reason = Timeout
8072026-03-10T20:08:05.602ZINFOtest_retry_backoff (BackgroundTasks): reconfigurator config not yet loaded; doing nothing background_task = tuf_repo_pruner
8082026-03-10T20:08:05.602ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = tuf_repo_pruner elapsed = 73.789\u{b5}s iteration = 1 status = {"reason":"reconfigurator config not yet loaded","status":"disabled"}
8092026-03-10T20:08:05.602ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
8102026-03-10T20:08:05.602ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = read_only_region_replacement_start resource = Database
8112026-03-10T20:08:05.602ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
8122026-03-10T20:08:05.602ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
8132026-03-10T20:08:05.603ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = alert_dispatcher iteration = 1 reason = Timeout
8142026-03-10T20:08:05.603ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database
8152026-03-10T20:08:05.603ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
8162026-03-10T20:08:05.603ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database result = Ok(())
8172026-03-10T20:08:05.603ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
8182026-03-10T20:08:05.603ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database
8192026-03-10T20:08:05.603ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
8202026-03-10T20:08:05.603ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(())
8212026-03-10T20:08:05.604ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = multicast_reconciler iteration = 1 reason = Timeout
8222026-03-10T20:08:05.604ZTRACtest_retry_backoff (BackgroundTasks): multicast group reconciler activating background_task = multicast_reconciler
8232026-03-10T20:08:05.604ZTRACtest_retry_backoff (BackgroundTasks): starting multicast reconciliation pass background_task = multicast_reconciler
8242026-03-10T20:08:05.604ZDEBGtest_retry_backoff (BackgroundTasks): skipping cache invalidation check: no inventory available background_task = multicast_reconciler
8252026-03-10T20:08:05.604ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8262026-03-10T20:08:05.604ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 1 reason = Timeout
8272026-03-10T20:08:05.604ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
8282026-03-10T20:08:05.604ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = fm_sitrep_loader iteration = 1 reason = Timeout
8292026-03-10T20:08:05.604ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Fleet
8302026-03-10T20:08:05.604ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = fm_sitrep_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8312026-03-10T20:08:05.604ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database
8322026-03-10T20:08:05.604ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_loader roles = RoleSet { roles: {} }
8332026-03-10T20:08:05.604ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database result = Ok(())
8342026-03-10T20:08:05.605ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = fm_sitrep_gc iteration = 1 reason = Timeout
8352026-03-10T20:08:05.605ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
8362026-03-10T20:08:05.605ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
8372026-03-10T20:08:05.605ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
8382026-03-10T20:08:05.605ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = probe_distributor iteration = 1 reason = Timeout
8392026-03-10T20:08:05.606ZINFOtest_retry_backoff (BackgroundTasks): distributing networking probes to sleds background_task = probe_distributor
8402026-03-10T20:08:05.606ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = probe_distributor resource = Fleet
8412026-03-10T20:08:05.606ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = probe_distributor resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8422026-03-10T20:08:05.606ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = probe_distributor resource = Database
8432026-03-10T20:08:05.606ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = probe_distributor roles = RoleSet { roles: {} }
8442026-03-10T20:08:05.606ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = probe_distributor resource = Database result = Ok(())
8452026-03-10T20:08:05.606ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = trust_quorum_manager iteration = 1 reason = Timeout
8462026-03-10T20:08:05.606ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = trust_quorum_manager bgtask = TrustQuorumManager resource = Fleet
8472026-03-10T20:08:05.606ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = trust_quorum_manager bgtask = TrustQuorumManager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8482026-03-10T20:08:05.606ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = trust_quorum_manager bgtask = TrustQuorumManager resource = Database
8492026-03-10T20:08:05.606ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = trust_quorum_manager bgtask = TrustQuorumManager roles = RoleSet { roles: {} }
8502026-03-10T20:08:05.606ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = trust_quorum_manager bgtask = TrustQuorumManager resource = Database result = Ok(())
8512026-03-10T20:08:05.607ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = attached_subnet_manager iteration = 1 reason = Timeout
8522026-03-10T20:08:05.607ZINFOtest_retry_backoff (BackgroundTasks): starting attached subnet manager work background_task = attached_subnet_manager
8532026-03-10T20:08:05.607ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8542026-03-10T20:08:05.609ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
8552026-03-10T20:08:05.609ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8562026-03-10T20:08:05.609ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database
8572026-03-10T20:08:05.609ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8582026-03-10T20:08:05.610ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8592026-03-10T20:08:05.611ZTRACtest_retry_backoff (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 12 a2 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:27349 req_id = 586bbaec-86a7-4569-b99b-310d72b68733
8602026-03-10T20:08:05.611ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4770,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:27349 req_id = 586bbaec-86a7-4569-b99b-310d72b68733
8612026-03-10T20:08:05.611ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4770, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:27349 req_id = 586bbaec-86a7-4569-b99b-310d72b68733
8622026-03-10T20:08:05.611ZTRACtest_retry_backoff (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 80 c5 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55603 req_id = b8cf2a7c-4e64-4731-8a7c-f96b203b4a4c
8632026-03-10T20:08:05.611ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32965,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:55603 req_id = b8cf2a7c-4e64-4731-8a7c-f96b203b4a4c
8642026-03-10T20:08:05.612ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32965, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55603 req_id = b8cf2a7c-4e64-4731-8a7c-f96b203b4a4c
8652026-03-10T20:08:05.612ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8662026-03-10T20:08:05.612ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8672026-03-10T20:08:05.612ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database
8682026-03-10T20:08:05.612ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8692026-03-10T20:08:05.612ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8702026-03-10T20:08:05.613ZWARNtest_retry_backoff (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = proto error: no records found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8712026-03-10T20:08:05.613ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 61.729365ms iteration = 1 status = {"error":"failed to read list of DNS servers: proto error: no records found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8722026-03-10T20:08:05.613ZWARNtest_retry_backoff (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = proto error: no records found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8732026-03-10T20:08:05.613ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 60.570176ms iteration = 1 status = {"error":"failed to read list of DNS servers: proto error: no records found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8742026-03-10T20:08:05.613ZTRACtest_retry_backoff (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 4d 21 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 M!..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:10323 req_id = 45aad745-92bf-410b-b67b-33e485c1b000
8752026-03-10T20:08:05.613ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19745,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:10323 req_id = 45aad745-92bf-410b-b67b-33e485c1b000
8762026-03-10T20:08:05.614ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19745, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:10323 req_id = 45aad745-92bf-410b-b67b-33e485c1b000
8772026-03-10T20:08:05.614ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 99 b2 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:23705 req_id = 4a95f840-df4e-4771-9ab8-573a116b910f
8782026-03-10T20:08:05.614ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39346,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:23705 req_id = 4a95f840-df4e-4771-9ab8-573a116b910f
8792026-03-10T20:08:05.614ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39346, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:23705 req_id = 4a95f840-df4e-4771-9ab8-573a116b910f
8802026-03-10T20:08:05.614ZTRACtest_retry_backoff (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: e9 4a 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .J..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:34076 req_id = 1095e40c-a813-455d-ac29-18f2ffd00db9
8812026-03-10T20:08:05.614ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59722,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:34076 req_id = 1095e40c-a813-455d-ac29-18f2ffd00db9
8822026-03-10T20:08:05.614ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59722, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:34076 req_id = 1095e40c-a813-455d-ac29-18f2ffd00db9
8832026-03-10T20:08:05.616ZERROtest_retry_backoff (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = multicast_reconciler error = proto error: no records found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8842026-03-10T20:08:05.616ZERROtest_retry_backoff (BackgroundTasks): failed to build DPD clients background_task = multicast_reconciler error = proto error: no records found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8852026-03-10T20:08:05.616ZERROtest_retry_backoff (BackgroundTasks): multicast RPW reconciliation pass completed with dataplane inconsistencies active_groups_verified = 0 background_task = multicast_reconciler dataplane_error_count = 1 external_groups_created = 0 external_groups_deleted = 0 member_state_transitions = 0 orphaned_members_cleaned = 0
8862026-03-10T20:08:05.616ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = multicast_reconciler elapsed = 12.188803ms iteration = 1 status = {"disabled":false,"empty_groups_marked":0,"errors":["failed to create multicast dataplane client: Internal Error: failed to build DPD clients"],"groups_created":0,"groups_deleted":0,"groups_verified":0,"members_deleted":0,"members_processed":0}
8872026-03-10T20:08:05.616ZERROtest_retry_backoff (BackgroundTasks): failed to resolve MGS addresses background_task = sp_ereport_ingester error = Resolve(ResolveError { kind: Proto(ProtoError { kind: NoRecordsFound { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, soa: None, ns: None, negative_ttl: None, response_code: ServFail, trusted: true, authorities: None } }) })
8882026-03-10T20:08:05.617ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 13.091475ms iteration = 1 status = {"disabled":false,"errors":["failed to resolve MGS addresses: proto error: no records found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"],"sps":[]}
8892026-03-10T20:08:05.617ZERROtest_retry_backoff (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = attached_subnet_manager error = proto error: no records found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8902026-03-10T20:08:05.617ZERROtest_retry_backoff (BackgroundTasks): failed to lookup Dendrite clients, will not be able to forward attachments to Dendrite background_task = attached_subnet_manager error = proto error: no records found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8912026-03-10T20:08:05.617ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = attached_subnet_manager resource = Database
8922026-03-10T20:08:05.617ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = attached_subnet_manager roles = RoleSet { roles: {} }
8932026-03-10T20:08:05.618ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = attached_subnet_manager resource = Database result = Ok(())
8942026-03-10T20:08:05.622ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8952026-03-10T20:08:05.623ZINFOtest_retry_backoff (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2026-03-10 19:58:05.553338644 UTC pruned = {}
8962026-03-10T20:08:05.623ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 70.151197ms iteration = 1 status = {"expiration":"2026-03-10T19:58:05.553338644Z","pruned":[]}
8972026-03-10T20:08:05.623ZTRACtest_retry_backoff (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8982026-03-10T20:08:05.625ZTRACtest_retry_backoff (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: b6 e6 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:33070 req_id = 92662978-467d-4b2c-a361-7aee8e79aa04
8992026-03-10T20:08:05.625ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46822,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:33070 req_id = 92662978-467d-4b2c-a361-7aee8e79aa04
9002026-03-10T20:08:05.625ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46822, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:33070 req_id = 92662978-467d-4b2c-a361-7aee8e79aa04
9012026-03-10T20:08:05.626ZTRACtest_retry_backoff (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: cb b4 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63983 req_id = 4dd085a7-6505-4b46-a2b7-a7c91d1deda3
9022026-03-10T20:08:05.626ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52148,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:63983 req_id = 4dd085a7-6505-4b46-a2b7-a7c91d1deda3
9032026-03-10T20:08:05.627ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52148, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63983 req_id = 4dd085a7-6505-4b46-a2b7-a7c91d1deda3
9042026-03-10T20:08:05.628ZERROtest_retry_backoff (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_garbage_collector error = proto error: no records found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9052026-03-10T20:08:05.628ZERROtest_retry_backoff (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_garbage_collector error = proto error: no records found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9062026-03-10T20:08:05.628ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = nat_garbage_collector elapsed = 70.651205ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: proto error: no records found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9072026-03-10T20:08:05.628ZERROtest_retry_backoff (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = proto error: no records found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9082026-03-10T20:08:05.628ZERROtest_retry_backoff (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = proto error: no records found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9092026-03-10T20:08:05.628ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 70.054831ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: proto error: no records found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9102026-03-10T20:08:05.696ZDEBGtest_retry_backoff (init_with_steps): Running step start_dendrite_switch0
9112026-03-10T20:08:05.696ZDEBGtest_retry_backoff: Starting Dendrite for switch0
9122026-03-10T20:08:05.762ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9132026-03-10T20:08:05.768ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9142026-03-10T20:08:05.768ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
9152026-03-10T20:08:05.768ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9162026-03-10T20:08:05.769ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
9172026-03-10T20:08:05.770ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9182026-03-10T20:08:05.774ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9192026-03-10T20:08:05.774ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Database
9202026-03-10T20:08:05.774ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9212026-03-10T20:08:05.774ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9222026-03-10T20:08:05.780ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9232026-03-10T20:08:05.788ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9242026-03-10T20:08:05.789ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database
9252026-03-10T20:08:05.789ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9262026-03-10T20:08:05.789ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database result = Ok(())
9272026-03-10T20:08:05.790ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9282026-03-10T20:08:05.794ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9292026-03-10T20:08:05.794ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database
9302026-03-10T20:08:05.794ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9312026-03-10T20:08:05.794ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9322026-03-10T20:08:05.795ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 236.536865ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
9332026-03-10T20:08:05.795ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9342026-03-10T20:08:05.800ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Inventory result = Ok(())
9352026-03-10T20:08:05.800ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database
9362026-03-10T20:08:05.800ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {} }
9372026-03-10T20:08:05.800ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database result = Ok(())
9382026-03-10T20:08:05.801ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9392026-03-10T20:08:05.806ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig result = Ok(())
9402026-03-10T20:08:05.806ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
9412026-03-10T20:08:05.806ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
9422026-03-10T20:08:05.806ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
9432026-03-10T20:08:05.807ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9442026-03-10T20:08:05.811ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9452026-03-10T20:08:05.811ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database
9462026-03-10T20:08:05.811ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9472026-03-10T20:08:05.811ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database result = Ok(())
9482026-03-10T20:08:05.815ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_replacement_driver resource = Database
9492026-03-10T20:08:05.815ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9502026-03-10T20:08:05.815ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9512026-03-10T20:08:05.820ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_replacement resource = Database
9522026-03-10T20:08:05.820ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9532026-03-10T20:08:05.820ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_replacement resource = Database result = Ok(())
9542026-03-10T20:08:05.822ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
9552026-03-10T20:08:05.822ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9562026-03-10T20:08:05.822ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9572026-03-10T20:08:05.826ZDEBGtest_retry_backoff (init_with_steps): Running step start_mgd_switch0
9582026-03-10T20:08:05.826ZDEBGtest_retry_backoff: Starting mgd for switch0
9592026-03-10T20:08:05.844ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9602026-03-10T20:08:05.848ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9612026-03-10T20:08:05.848ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet
9622026-03-10T20:08:05.848ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9632026-03-10T20:08:05.848ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
9642026-03-10T20:08:05.848ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9652026-03-10T20:08:05.848ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9662026-03-10T20:08:05.853ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9672026-03-10T20:08:05.861ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9682026-03-10T20:08:05.861ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Database
9692026-03-10T20:08:05.861ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9702026-03-10T20:08:05.861ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Database result = Ok(())
9712026-03-10T20:08:05.863ZINFOtest_retry_backoff (dropshot_internal): accepted connection local_addr = 127.0.0.1:63060 remote_addr = 127.0.0.1:48817
9722026-03-10T20:08:05.864ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:48817 req_id = e95d0af3-a6d1-478d-9a87-3a468429b943 uri = /nat/ipv4/changeset/0?limit=100
9732026-03-10T20:08:05.864ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:48817 req_id = e95d0af3-a6d1-478d-9a87-3a468429b943 uri = /nat/ipv4/changeset/0?limit=100 version = 1.0.0
9742026-03-10T20:08:05.865ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:48817 req_id = e95d0af3-a6d1-478d-9a87-3a468429b943 resource = Database type = user_builtin uri = /nat/ipv4/changeset/0?limit=100 user_builtin_id = 001de000-05e4-4000-8000-000000000002
9752026-03-10T20:08:05.865ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:48817 req_id = e95d0af3-a6d1-478d-9a87-3a468429b943 roles = RoleSet { roles: {} } type = user_builtin uri = /nat/ipv4/changeset/0?limit=100 user_builtin_id = 001de000-05e4-4000-8000-000000000002
9762026-03-10T20:08:05.865ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:48817 req_id = e95d0af3-a6d1-478d-9a87-3a468429b943 resource = Database result = Ok(()) type = user_builtin uri = /nat/ipv4/changeset/0?limit=100 user_builtin_id = 001de000-05e4-4000-8000-000000000002
9772026-03-10T20:08:05.867ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 268.927196ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9782026-03-10T20:08:05.884ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9792026-03-10T20:08:05.886ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Fleet result = Ok(())
9802026-03-10T20:08:05.886ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database
9812026-03-10T20:08:05.886ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_loader roles = RoleSet { roles: {} }
9822026-03-10T20:08:05.886ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database result = Ok(())
9832026-03-10T20:08:05.911ZDEBGtest_retry_backoff: mgd port is 52170
9842026-03-10T20:08:05.911ZDEBGtest_retry_backoff (init_with_steps): Running step record_switch_dns
9852026-03-10T20:08:05.911ZDEBGtest_retry_backoff: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
9862026-03-10T20:08:05.911ZDEBGtest_retry_backoff (init_with_steps): Running step start_sled1
9872026-03-10T20:08:05.912ZINFOtest_retry_backoff (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
9882026-03-10T20:08:05.976ZINFOtest_retry_backoff (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
9892026-03-10T20:08:05.976ZINFOtest_retry_backoff (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
9902026-03-10T20:08:05.977ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_step resource = Database
9912026-03-10T20:08:05.977ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9922026-03-10T20:08:05.978ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9932026-03-10T20:08:05.979ZINFOtest_retry_backoff (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator num_receivers = 0
9942026-03-10T20:08:05.979ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 376.228201ms iteration = 1 status = {"by_rx":{},"error":null}
9952026-03-10T20:08:05.980ZINFOtest_retry_backoff (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpMl1Ylu/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
9962026-03-10T20:08:05.980ZDEBGtest_retry_backoff (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpMl1Ylu/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
9972026-03-10T20:08:05.980ZINFOtest_retry_backoff (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpIgjk6y/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
9982026-03-10T20:08:05.980ZDEBGtest_retry_backoff (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpIgjk6y/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
9992026-03-10T20:08:06.059ZDEBGtest_retry_backoff (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:47851 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10002026-03-10T20:08:06.059ZINFOtest_retry_backoff (dropshot (Repo Depot)): listening local_addr = [::1]:47851 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10012026-03-10T20:08:06.060ZDEBGtest_retry_backoff (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:47851 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10022026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10032026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10042026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10052026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10062026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10072026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10082026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /chicken-switch/destroy-orphaned-datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 2.0.0
10092026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /chicken-switch/destroy-orphaned-datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 3.0.0
10102026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /debug/switch-zone-policy sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 3.0.0
10112026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /debug/switch-zone-policy sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 3.0.0
10122026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 21.0.0
10132026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10142026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 24.0.0
10152026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 22.0.0 to 24.0.0
10162026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 16.0.0 to 22.0.0
10172026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 14.0.0 to 16.0.0
10182026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 12.0.0 to 14.0.0
10192026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 11.0.0 to 12.0.0
10202026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 10.0.0 to 11.0.0
10212026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 4.0.0 to 10.0.0
10222026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 4.0.0
10232026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = DELETE path = /local-storage sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 17.0.0
10242026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /local-storage sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 17.0.0
10252026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = DELETE path = /local-storage/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 9.0.0 to 17.0.0
10262026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /local-storage/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 9.0.0 to 17.0.0
10272026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 20.0.0 to 23.0.0
10282026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 20.0.0
10292026-03-10T20:08:06.066ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 26.0.0
10302026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 25.0.0 to 26.0.0
10312026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 20.0.0 to 25.0.0
10322026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 20.0.0
10332026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 14.0.0
10342026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 11.0.0 to 14.0.0
10352026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 10.0.0 to 11.0.0
10362026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 4.0.0 to 10.0.0
10372026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 4.0.0
10382026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /probes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 10.0.0
10392026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /probes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 6.0.0 to 10.0.0
10402026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /rot/{rot}/attest sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 19.0.0
10412026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /rot/{rot}/certificate-chain sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 19.0.0
10422026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /rot/{rot}/measurement-log sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 19.0.0
10432026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10442026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 8.0.0
10452026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10462026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10472026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/health-check sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10482026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10492026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10502026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10512026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10522026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10532026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10542026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10552026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10562026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10572026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10582026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10592026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10602026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10612026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10622026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10632026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10642026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10652026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/finalize sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10662026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10672026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10682026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/transfer sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10692026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 20.0.0
10702026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 20.0.0
10712026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /trust-quorum/commit sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 13.0.0
10722026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /trust-quorum/configuration sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 13.0.0
10732026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /trust-quorum/coordinator-status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 13.0.0
10742026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /trust-quorum/network-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 15.0.0
10752026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /trust-quorum/network-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 15.0.0
10762026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /trust-quorum/prepare-and-commit sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 13.0.0
10772026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /trust-quorum/proxy/commit sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 13.0.0
10782026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /trust-quorum/proxy/prepare-and-commit sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 13.0.0
10792026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /trust-quorum/proxy/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 13.0.0
10802026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /trust-quorum/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 15.0.0
10812026-03-10T20:08:06.067ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /trust-quorum/upgrade sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 13.0.0
10822026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10832026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10842026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10852026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10862026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 18.0.0
10872026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 17.0.0 to 18.0.0
10882026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 11.0.0 to 17.0.0
10892026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 10.0.0 to 11.0.0
10902026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 9.0.0 to 10.0.0
10912026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = from 7.0.0 to 9.0.0
10922026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 7.0.0
10932026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = DELETE path = /vmms/{propolis_id}/attached-subnets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 18.0.0
10942026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /vmms/{propolis_id}/attached-subnets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 18.0.0
10952026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id}/attached-subnets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 18.0.0
10962026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = DELETE path = /vmms/{propolis_id}/attached-subnets/{subnet} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 18.0.0
10972026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10982026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10992026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11002026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = DELETE path = /vmms/{propolis_id}/multicast-group sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 7.0.0
11012026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id}/multicast-group sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 7.0.0
11022026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11032026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11042026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11052026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11062026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11072026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all starting from 10.0.0
11082026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all ending with 10.0.0
11092026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11102026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11112026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11122026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11132026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11142026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11152026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11162026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11172026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11182026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11192026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:45969 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
11202026-03-10T20:08:06.068ZINFOtest_retry_backoff (dropshot): listening local_addr = [::1]:45969 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11212026-03-10T20:08:06.068ZDEBGtest_retry_backoff (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:45969 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11222026-03-10T20:08:06.068ZDEBGtest_retry_backoff (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11232026-03-10T20:08:06.068ZDEBGtest_retry_backoff (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:63060/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
11242026-03-10T20:08:06.069ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet
11252026-03-10T20:08:06.069ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = fm_sitrep_gc resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11262026-03-10T20:08:06.069ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
11272026-03-10T20:08:06.069ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
11282026-03-10T20:08:06.069ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
11292026-03-10T20:08:06.070ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_start resource = Database
11302026-03-10T20:08:06.070ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
11312026-03-10T20:08:06.071ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
11322026-03-10T20:08:06.071ZINFOtest_retry_backoff (SagaRecoverer): listed in-progress sagas count = 0
11332026-03-10T20:08:06.071ZINFOtest_retry_backoff (SagaQuiesceHandle): saga recovery pass finished blueprint_id = None generation = 1
11342026-03-10T20:08:06.071ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 501.222348ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
11352026-03-10T20:08:06.072ZDEBGtest_retry_backoff (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
11362026-03-10T20:08:06.072ZINFOtest_retry_backoff (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
11372026-03-10T20:08:06.072ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 502.088941ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
11382026-03-10T20:08:06.072ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 471.053369ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
11392026-03-10T20:08:06.072ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11402026-03-10T20:08:06.075ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11412026-03-10T20:08:06.075ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Fleet
11422026-03-10T20:08:06.075ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11432026-03-10T20:08:06.075ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database
11442026-03-10T20:08:06.075ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11452026-03-10T20:08:06.075ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11462026-03-10T20:08:06.076ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 476.312665ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
11472026-03-10T20:08:06.076ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 473.833446ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
11482026-03-10T20:08:06.076ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database
11492026-03-10T20:08:06.076ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11502026-03-10T20:08:06.076ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database result = Ok(())
11512026-03-10T20:08:06.077ZINFOtest_retry_backoff (dropshot_internal): accepted connection local_addr = 127.0.0.1:63060 remote_addr = 127.0.0.1:52204
11522026-03-10T20:08:06.077ZTRACtest_retry_backoff (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
11532026-03-10T20:08:06.077ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_reincarnation resource = Database
11542026-03-10T20:08:06.077ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
11552026-03-10T20:08:06.077ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_reincarnation resource = Database result = Ok(())
11562026-03-10T20:08:06.078ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = probe_distributor roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11572026-03-10T20:08:06.081ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = probe_distributor resource = Fleet result = Ok(())
11582026-03-10T20:08:06.081ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = probe_distributor resource = Fleet
11592026-03-10T20:08:06.081ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = probe_distributor resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11602026-03-10T20:08:06.081ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = probe_distributor resource = Database
11612026-03-10T20:08:06.081ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = probe_distributor roles = RoleSet { roles: {} }
11622026-03-10T20:08:06.081ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = probe_distributor resource = Database result = Ok(())
11632026-03-10T20:08:06.083ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:52204 req_id = bbbf21c4-b3dd-4b14-b110-50f3fb94a400 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
11642026-03-10T20:08:06.083ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:52204 req_id = bbbf21c4-b3dd-4b14-b110-50f3fb94a400 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229 version = 1.0.0
11652026-03-10T20:08:06.085ZINFOtest_retry_backoff (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
11662026-03-10T20:08:06.089ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11672026-03-10T20:08:06.096ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
11682026-03-10T20:08:06.097ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 534.235449ms iteration = 1 status = {}
11692026-03-10T20:08:06.098ZWARNtest_retry_backoff (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
11702026-03-10T20:08:06.098ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 536.404788ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
11712026-03-10T20:08:06.098ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = inventory_loader elapsed = 538.342958ms iteration = 1 status = "NoCollections"
11722026-03-10T20:08:06.099ZDEBGtest_retry_backoff (BackgroundTasks): reconfigurator config load complete background_task = reconfigurator_config_watcher switches = Loaded(ReconfiguratorConfigView { version: 1, config: ReconfiguratorConfig { planner_enabled: false, planner_config: PlannerConfig { add_zones_with_mupdate_override: false }, tuf_repo_pruner_enabled: true }, time_modified: 2026-03-10T20:08:05.458980Z })
11732026-03-10T20:08:06.099ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = reconfigurator_config_watcher elapsed = 537.969912ms iteration = 1 status = {"config_updated":true}
11742026-03-10T20:08:06.099ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = blueprint_planner iteration = 2 reason = Dependency
11752026-03-10T20:08:06.099ZDEBGtest_retry_backoff (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
11762026-03-10T20:08:06.099ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 104.567\u{b5}s iteration = 2 status = "Disabled"
11772026-03-10T20:08:06.099ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = tuf_repo_pruner iteration = 2 reason = Dependency
11782026-03-10T20:08:06.099ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Fleet
11792026-03-10T20:08:06.099ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = tuf_repo_pruner resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11802026-03-10T20:08:06.099ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database
11812026-03-10T20:08:06.099ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_repo_pruner roles = RoleSet { roles: {} }
11822026-03-10T20:08:06.100ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database result = Ok(())
11832026-03-10T20:08:06.101ZDEBGtest_retry_backoff (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11842026-03-10T20:08:06.101ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11852026-03-10T20:08:06.101ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f6 9c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:6774 req_id = ccb0e061-d6b7-4011-b9c7-b9d1de1bd117
11862026-03-10T20:08:06.101ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63132,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:6774 req_id = ccb0e061-d6b7-4011-b9c7-b9d1de1bd117
11872026-03-10T20:08:06.102ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63132, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:6774 req_id = ccb0e061-d6b7-4011-b9c7-b9d1de1bd117
11882026-03-10T20:08:06.102ZWARNtest_retry_backoff (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: proto error: no records found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
11892026-03-10T20:08:06.102ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 542.96794ms iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: proto error: no records found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
11902026-03-10T20:08:06.111ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = DnsConfig
11912026-03-10T20:08:06.111ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11922026-03-10T20:08:06.111ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database
11932026-03-10T20:08:06.111ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11942026-03-10T20:08:06.112ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database result = Ok(())
11952026-03-10T20:08:06.117ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
11962026-03-10T20:08:06.117ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11972026-03-10T20:08:06.118ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11982026-03-10T20:08:06.125ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11992026-03-10T20:08:06.130ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12002026-03-10T20:08:06.130ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
12012026-03-10T20:08:06.130ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12022026-03-10T20:08:06.130ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12032026-03-10T20:08:06.134ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = fm_sitrep_loader elapsed = 529.951295ms iteration = 1 status = "NoSitrep"
12042026-03-10T20:08:06.144ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 279791 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:48817 req_id = e95d0af3-a6d1-478d-9a87-3a468429b943 response_code = 200 uri = /nat/ipv4/changeset/0?limit=100
12052026-03-10T20:08:06.146ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = trust_quorum_manager bgtask = TrustQuorumManager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12062026-03-10T20:08:06.150ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = trust_quorum_manager bgtask = TrustQuorumManager resource = Fleet result = Ok(())
12072026-03-10T20:08:06.150ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = trust_quorum_manager bgtask = TrustQuorumManager resource = Database
12082026-03-10T20:08:06.150ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = trust_quorum_manager bgtask = TrustQuorumManager roles = RoleSet { roles: {} }
12092026-03-10T20:08:06.150ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = trust_quorum_manager bgtask = TrustQuorumManager resource = Database result = Ok(())
12102026-03-10T20:08:06.151ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Fleet
12112026-03-10T20:08:06.151ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12122026-03-10T20:08:06.151ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Database
12132026-03-10T20:08:06.151ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12142026-03-10T20:08:06.152ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12152026-03-10T20:08:06.159ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 560.173606ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
12162026-03-10T20:08:06.169ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = region_replacement elapsed = 604.070815ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
12172026-03-10T20:08:06.170ZWARNtest_retry_backoff (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
12182026-03-10T20:08:06.170ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 618.01588ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
12192026-03-10T20:08:06.170ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12202026-03-10T20:08:06.181ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
12212026-03-10T20:08:06.181ZTRACtest_retry_backoff (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = alert_dispatcher
12222026-03-10T20:08:06.181ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database
12232026-03-10T20:08:06.181ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
12242026-03-10T20:08:06.182ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database result = Ok(())
12252026-03-10T20:08:06.189ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = probe_distributor roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12262026-03-10T20:08:06.192ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = probe_distributor resource = Fleet result = Ok(())
12272026-03-10T20:08:06.192ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = probe_distributor resource = Database
12282026-03-10T20:08:06.192ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = probe_distributor roles = RoleSet { roles: {} }
12292026-03-10T20:08:06.192ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = probe_distributor resource = Database result = Ok(())
12302026-03-10T20:08:06.193ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 627.417934ms iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12312026-03-10T20:08:06.196ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12322026-03-10T20:08:06.213ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12332026-03-10T20:08:06.213ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database
12342026-03-10T20:08:06.213ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12352026-03-10T20:08:06.214ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12362026-03-10T20:08:06.215ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12372026-03-10T20:08:06.215ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12382026-03-10T20:08:06.215ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
12392026-03-10T20:08:06.215ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12402026-03-10T20:08:06.215ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12412026-03-10T20:08:06.230ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_repo_pruner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12422026-03-10T20:08:06.234ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Fleet result = Ok(())
12432026-03-10T20:08:06.234ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database
12442026-03-10T20:08:06.234ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_repo_pruner roles = RoleSet { roles: {} }
12452026-03-10T20:08:06.234ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database result = Ok(())
12462026-03-10T20:08:06.240ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12472026-03-10T20:08:06.246ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
12482026-03-10T20:08:06.246ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Database
12492026-03-10T20:08:06.246ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12502026-03-10T20:08:06.246ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12512026-03-10T20:08:06.288ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12522026-03-10T20:08:06.291ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
12532026-03-10T20:08:06.291ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12542026-03-10T20:08:06.292ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12552026-03-10T20:08:06.294ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12562026-03-10T20:08:06.301ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
12572026-03-10T20:08:06.301ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database
12582026-03-10T20:08:06.301ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12592026-03-10T20:08:06.301ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database result = Ok(())
12602026-03-10T20:08:06.302ZINFOtest_retry_backoff (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
12612026-03-10T20:08:06.302ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 735.941624ms iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
12622026-03-10T20:08:06.330ZDEBGtest_retry_backoff (BackgroundTasks): listed attached subnets background_task = attached_subnet_manager n_subnets = 0
12632026-03-10T20:08:06.335ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = attached_subnet_manager elapsed = 728.532574ms iteration = 1 status = {"db_error":null,"dendrite":{},"sled":{}}
12642026-03-10T20:08:06.343ZTRACtest_retry_backoff (BackgroundTasks): webhook dispatching completed successfully background_task = alert_dispatcher events_dispatched = 0 events_without_receivers = 0
12652026-03-10T20:08:06.343ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 740.650588ms iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"237.0.0","globs_reprocessed":{},"no_receivers":[]}
12662026-03-10T20:08:06.352ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12672026-03-10T20:08:06.355ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet result = Ok(())
12682026-03-10T20:08:06.356ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_step resource = Database
12692026-03-10T20:08:06.356ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12702026-03-10T20:08:06.356ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12712026-03-10T20:08:06.365ZWARNtest_retry_backoff (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
12722026-03-10T20:08:06.365ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 816.685612ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
12732026-03-10T20:08:06.374ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Fleet
12742026-03-10T20:08:06.374ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = tuf_repo_pruner resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12752026-03-10T20:08:06.374ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database
12762026-03-10T20:08:06.374ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_repo_pruner roles = RoleSet { roles: {} }
12772026-03-10T20:08:06.375ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database result = Ok(())
12782026-03-10T20:08:06.385ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12792026-03-10T20:08:06.385ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
12802026-03-10T20:08:06.385ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12812026-03-10T20:08:06.386ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12822026-03-10T20:08:06.387ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 818.098226ms iteration = 1 status = {}
12832026-03-10T20:08:06.411ZINFOtest_retry_backoff (BackgroundTasks): Loaded 0 active trust quorum configurations from database background_task = trust_quorum_manager bgtask = TrustQuorumManager
12842026-03-10T20:08:06.411ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = trust_quorum_manager elapsed = 804.524456ms iteration = 1 status = {"PerRackStatus":{"errors":[],"statuses":[]}}
12852026-03-10T20:08:06.411ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
12862026-03-10T20:08:06.411ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
12872026-03-10T20:08:06.411ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
12882026-03-10T20:08:06.412ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_repo_pruner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12892026-03-10T20:08:06.416ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Fleet result = Ok(())
12902026-03-10T20:08:06.416ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database
12912026-03-10T20:08:06.416ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_repo_pruner roles = RoleSet { roles: {} }
12922026-03-10T20:08:06.417ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database result = Ok(())
12932026-03-10T20:08:06.418ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet
12942026-03-10T20:08:06.418ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = fm_sitrep_gc resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12952026-03-10T20:08:06.418ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
12962026-03-10T20:08:06.418ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
12972026-03-10T20:08:06.418ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
12982026-03-10T20:08:06.419ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12992026-03-10T20:08:06.425ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13002026-03-10T20:08:06.425ZTRACtest_retry_backoff (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
13012026-03-10T20:08:06.425ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
13022026-03-10T20:08:06.425ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13032026-03-10T20:08:06.425ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
13042026-03-10T20:08:06.425ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13052026-03-10T20:08:06.425ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13062026-03-10T20:08:06.431ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 831.055176ms iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
13072026-03-10T20:08:06.460ZINFOtest_retry_backoff (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
13082026-03-10T20:08:06.460ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 897.428409ms iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
13092026-03-10T20:08:06.482ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = TargetReleaseConfig
13102026-03-10T20:08:06.482ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = tuf_repo_pruner resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13112026-03-10T20:08:06.482ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database
13122026-03-10T20:08:06.482ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_repo_pruner roles = RoleSet { roles: {} }
13132026-03-10T20:08:06.482ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database result = Ok(())
13142026-03-10T20:08:06.499ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Fleet
13152026-03-10T20:08:06.499ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13162026-03-10T20:08:06.499ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database
13172026-03-10T20:08:06.499ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
13182026-03-10T20:08:06.499ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
13192026-03-10T20:08:06.505ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_repo_pruner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13202026-03-10T20:08:06.509ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = TargetReleaseConfig result = Ok(())
13212026-03-10T20:08:06.509ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database
13222026-03-10T20:08:06.509ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_repo_pruner roles = RoleSet { roles: {} }
13232026-03-10T20:08:06.510ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_repo_pruner resource = Database result = Ok(())
13242026-03-10T20:08:06.511ZINFOtest_retry_backoff (BackgroundTasks): finished distributing probes to sleds background_task = probe_distributor n_pushed_probes = 0
13252026-03-10T20:08:06.511ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = probe_distributor elapsed = 905.180728ms iteration = 1 status = {"errors":[],"probes_by_sled":{}}
13262026-03-10T20:08:06.511ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Dependency
13272026-03-10T20:08:06.511ZINFOtest_retry_backoff (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
13282026-03-10T20:08:06.511ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet
13292026-03-10T20:08:06.511ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13302026-03-10T20:08:06.511ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
13312026-03-10T20:08:06.511ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13322026-03-10T20:08:06.511ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13332026-03-10T20:08:06.512ZWARNtest_retry_backoff (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
13342026-03-10T20:08:06.512ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 958.063663ms iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
13352026-03-10T20:08:06.512ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13362026-03-10T20:08:06.512ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
13372026-03-10T20:08:06.512ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13382026-03-10T20:08:06.512ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13392026-03-10T20:08:06.522ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13402026-03-10T20:08:06.524ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet result = Ok(())
13412026-03-10T20:08:06.542ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13422026-03-10T20:08:06.542ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
13432026-03-10T20:08:06.542ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13442026-03-10T20:08:06.542ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13452026-03-10T20:08:06.549ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13462026-03-10T20:08:06.555ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
13472026-03-10T20:08:06.555ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet
13482026-03-10T20:08:06.555ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13492026-03-10T20:08:06.555ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
13502026-03-10T20:08:06.555ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13512026-03-10T20:08:06.556ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13522026-03-10T20:08:06.561ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Fleet
13532026-03-10T20:08:06.561ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13542026-03-10T20:08:06.561ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database
13552026-03-10T20:08:06.561ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13562026-03-10T20:08:06.562ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database result = Ok(())
13572026-03-10T20:08:06.563ZINFOtest_retry_backoff (BackgroundTasks): tuf_repo_prune decision background_task = tuf_repo_pruner status = TufRepoPrunerDetails { nkeep_recent_releases: 3, nkeep_recent_uploads: 2, repos_keep_target_release: {}, repos_keep_recent_uploads: {}, repo_prune: None, other_repos_eligible_to_prune: {}, warnings: [] }
13582026-03-10T20:08:06.563ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = tuf_repo_pruner elapsed = 463.745675ms iteration = 2 status = {"nkeep_recent_releases":3,"nkeep_recent_uploads":2,"other_repos_eligible_to_prune":[],"repo_prune":null,"repos_keep_recent_uploads":[],"repos_keep_target_release":[],"status":"enabled","warnings":[]}
13592026-03-10T20:08:06.596ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13602026-03-10T20:08:06.609ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13612026-03-10T20:08:06.609ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
13622026-03-10T20:08:06.609ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13632026-03-10T20:08:06.609ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13642026-03-10T20:08:06.639ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
13652026-03-10T20:08:06.639ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Inventory
13662026-03-10T20:08:06.639ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13672026-03-10T20:08:06.639ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
13682026-03-10T20:08:06.640ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
13692026-03-10T20:08:06.640ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
13702026-03-10T20:08:06.641ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 558068 local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:52204 req_id = bbbf21c4-b3dd-4b14-b110-50f3fb94a400 response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
13712026-03-10T20:08:06.642ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13722026-03-10T20:08:06.646ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13732026-03-10T20:08:06.646ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Fleet
13742026-03-10T20:08:06.646ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13752026-03-10T20:08:06.646ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database
13762026-03-10T20:08:06.646ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13772026-03-10T20:08:06.646ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database result = Ok(())
13782026-03-10T20:08:06.647ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13792026-03-10T20:08:06.651ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
13802026-03-10T20:08:06.659ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
13812026-03-10T20:08:06.661ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13822026-03-10T20:08:06.662ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13832026-03-10T20:08:06.664ZTRACtest_retry_backoff (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
13842026-03-10T20:08:06.664ZINFOtest_retry_backoff (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
13852026-03-10T20:08:06.664ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 1.096814937s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
13862026-03-10T20:08:06.667ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
13872026-03-10T20:08:06.667ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13882026-03-10T20:08:06.668ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13892026-03-10T20:08:06.669ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13902026-03-10T20:08:06.681ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Inventory result = Ok(())
13912026-03-10T20:08:06.681ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
13922026-03-10T20:08:06.681ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
13932026-03-10T20:08:06.682ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
13942026-03-10T20:08:06.684ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13952026-03-10T20:08:06.688ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13962026-03-10T20:08:06.688ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database
13972026-03-10T20:08:06.688ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13982026-03-10T20:08:06.688ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database result = Ok(())
13992026-03-10T20:08:06.691ZDEBGtest_retry_backoff (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:63060/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "bbbf21c4-b3dd-4b14-b110-50f3fb94a400", "date": "Tue, 10 Mar 2026 20:08:06 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
14002026-03-10T20:08:06.692ZDEBGtest_retry_backoff (init_with_steps): Running step configure_boundary_ntp
14012026-03-10T20:08:06.692ZDEBGtest_retry_backoff (init_with_steps): Running step start_crucible_pantry
14022026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14032026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14042026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14052026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14062026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14072026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14082026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14092026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14102026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14112026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14122026-03-10T20:08:06.693ZDEBGtest_retry_backoff (dropshot): registered endpoint kind = pantry local_addr = [::1]:36870 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
14132026-03-10T20:08:06.693ZINFOtest_retry_backoff (dropshot): listening kind = pantry local_addr = [::1]:36870 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
14142026-03-10T20:08:06.694ZDEBGtest_retry_backoff (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:36870 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
14152026-03-10T20:08:06.694ZINFOtest_retry_backoff (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:36870 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
14162026-03-10T20:08:06.694ZDEBGtest_retry_backoff (init_with_steps): Running step populate_internal_dns
14172026-03-10T20:08:06.694ZDEBGtest_retry_backoff: Populating Internal DNS
14182026-03-10T20:08:06.740ZINFOtest_retry_backoff: DNS population: DnsConfigParams { generation: Generation( 1, ), serial: 1, time_created: 2026-03-10T20:08:06.739629439Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 64860, target: "81796aa6-4ecc-4d56-a811-dc112da8da6c.host.control-plane.oxide.internal", }, ), ], "5c201080-cc41-4511-8008-eb7a16cba8eb.host": [ Aaaa( ::1, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "3d27cdef-8860-4b0a-8825-636610fd4a26.host.control-plane.oxide.internal", }, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "3d27cdef-8860-4b0a-8825-636610fd4a26.host.control-plane.oxide.internal", }, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "3d27cdef-8860-4b0a-8825-636610fd4a26.host.control-plane.oxide.internal", }, ), ], "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 43524, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 36870, target: "3b196367-678f-4598-9fb6-cf1a4563fd74.host.control-plane.oxide.internal", }, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 51524, target: "3d27cdef-8860-4b0a-8825-636610fd4a26.host.control-plane.oxide.internal", }, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 52170, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "61d3324b-3a36-494c-95ab-02b65214832c.host": [ Aaaa( ::1, ), ], "913233fe-92a8-4635-9572-183f495429c4.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "81796aa6-4ecc-4d56-a811-dc112da8da6c.host": [ Aaaa( ::1, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 42738, target: "f189113c-85eb-4d6a-b2fc-46ba3de160e6.host.control-plane.oxide.internal", }, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 63060, target: "913233fe-92a8-4635-9572-183f495429c4.host.control-plane.oxide.internal", }, ), ], "_nexus-lockstep._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 50070, target: "913233fe-92a8-4635-9572-183f495429c4.host.control-plane.oxide.internal", }, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 41467, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 61901, target: "61d3324b-3a36-494c-95ab-02b65214832c.host.control-plane.oxide.internal", }, ), ], "boundary-ntp": [ Aaaa( ::1, ), ], "_boundary-ntp._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 123, target: "5c201080-cc41-4511-8008-eb7a16cba8eb.host.control-plane.oxide.internal", }, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "3b196367-678f-4598-9fb6-cf1a4563fd74.host": [ Aaaa( ::1, ), ], "3d27cdef-8860-4b0a-8825-636610fd4a26.host": [ Aaaa( ::1, ), ], "f189113c-85eb-4d6a-b2fc-46ba3de160e6.host": [ Aaaa( ::1, ), ], "ns1": [ Aaaa( ::1, ), ], "@": [ Ns( "ns1.control-plane.oxide.internal", ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 47851, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], }, }, ], }
14192026-03-10T20:08:06.740ZDEBGtest_retry_backoff: client request body = Some(Body) method = PUT uri = http://[::1]:64860/config
14202026-03-10T20:08:06.741ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
14212026-03-10T20:08:06.741ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14222026-03-10T20:08:06.741ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14232026-03-10T20:08:06.779ZINFOtest_retry_backoff (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
14242026-03-10T20:08:06.779ZDEBGtest_retry_backoff (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:45969/vpc-routes
14252026-03-10T20:08:06.779ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14262026-03-10T20:08:06.783ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
14272026-03-10T20:08:06.783ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database
14282026-03-10T20:08:06.783ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
14292026-03-10T20:08:06.783ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
14302026-03-10T20:08:06.784ZDEBGtest_retry_backoff (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
14312026-03-10T20:08:06.784ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
14322026-03-10T20:08:06.785ZINFOtest_retry_backoff (http): accepted connection kind = dns local_addr = [::1]:64860 remote_addr = [::1]:47625
14332026-03-10T20:08:06.785ZINFOtest_retry_backoff (dropshot): accepted connection local_addr = [::1]:45969 remote_addr = [::1]:60960 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
14342026-03-10T20:08:06.787ZTRACtest_retry_backoff (dropshot): incoming request local_addr = [::1]:45969 method = GET remote_addr = [::1]:60960 req_id = 2a0f963e-f804-499d-af9c-c29b169663c3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
14352026-03-10T20:08:06.787ZDEBGtest_retry_backoff (dropshot): determined request API version local_addr = [::1]:45969 method = GET remote_addr = [::1]:60960 req_id = 2a0f963e-f804-499d-af9c-c29b169663c3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes version = 27.0.0
14362026-03-10T20:08:06.788ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ba ab 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:3491 req_id = 56c40866-e379-475f-95a4-8c8e463af1e5
14372026-03-10T20:08:06.788ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47787,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:3491 req_id = 56c40866-e379-475f-95a4-8c8e463af1e5
14382026-03-10T20:08:06.788ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47787, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:3491 req_id = 56c40866-e379-475f-95a4-8c8e463af1e5
14392026-03-10T20:08:06.788ZINFOtest_retry_backoff (dropshot): request completed latency_us = 934 local_addr = [::1]:45969 method = GET remote_addr = [::1]:60960 req_id = 2a0f963e-f804-499d-af9c-c29b169663c3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
14402026-03-10T20:08:06.789ZWARNtest_retry_backoff (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: proto error: no records found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
14412026-03-10T20:08:06.789ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 149.475889ms iteration = 2 status = {"error":"failed to collect inventory: looking up MGS addresses: proto error: no records found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
14422026-03-10T20:08:06.789ZTRACtest_retry_backoff (http): incoming request kind = dns local_addr = [::1]:64860 method = PUT remote_addr = [::1]:47625 req_id = 93175aa7-e7d5-416b-be41-abcdb609651e uri = /config
14432026-03-10T20:08:06.789ZDEBGtest_retry_backoff (http): determined request API version kind = dns local_addr = [::1]:64860 method = PUT remote_addr = [::1]:47625 req_id = 93175aa7-e7d5-416b-be41-abcdb609651e uri = /config version = 2.0.0
14442026-03-10T20:08:06.790ZDEBGtest_retry_backoff (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:45969/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a0f963e-f804-499d-af9c-c29b169663c3", "content-length": "2", "date": "Tue, 10 Mar 2026 20:08:06 GMT"} })
14452026-03-10T20:08:06.790ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
14462026-03-10T20:08:06.790ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14472026-03-10T20:08:06.791ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14482026-03-10T20:08:06.792ZINFOtest_retry_backoff (store): attempting generation update new_generation = 1 req_id = 93175aa7-e7d5-416b-be41-abcdb609651e
14492026-03-10T20:08:06.792ZINFOtest_retry_backoff (store): pruning trees for generations newer than 0
14502026-03-10T20:08:06.792ZDEBGtest_retry_backoff (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
14512026-03-10T20:08:06.793ZDEBGtest_retry_backoff (store): updating current config
14522026-03-10T20:08:06.793ZDEBGtest_retry_backoff (store): flushing default tree
14532026-03-10T20:08:06.793ZINFOtest_retry_backoff (store): pruning trees for generations older than 1 keep = 3
14542026-03-10T20:08:06.793ZINFOtest_retry_backoff (store): updated generation new_generation = 1 req_id = 93175aa7-e7d5-416b-be41-abcdb609651e
14552026-03-10T20:08:06.794ZINFOtest_retry_backoff (http): request completed kind = dns latency_us = 4404 local_addr = [::1]:64860 method = PUT remote_addr = [::1]:47625 req_id = 93175aa7-e7d5-416b-be41-abcdb609651e response_code = 204 uri = /config
14562026-03-10T20:08:06.794ZDEBGtest_retry_backoff: client response result = Ok(Response { url: "http://[::1]:64860/config", status: 204, headers: {"x-request-id": "93175aa7-e7d5-416b-be41-abcdb609651e", "date": "Tue, 10 Mar 2026 20:08:06 GMT"} })
14572026-03-10T20:08:06.794ZDEBGtest_retry_backoff (init_with_steps): Running step configure_sled_agents
14582026-03-10T20:08:06.856ZDEBGtest_retry_backoff: client request body = Some(Body) method = PUT uri = http://[::1]:45969/omicron-config
14592026-03-10T20:08:06.858ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
14602026-03-10T20:08:06.858ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14612026-03-10T20:08:06.859ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14622026-03-10T20:08:06.860ZINFOtest_retry_backoff (dropshot): accepted connection local_addr = [::1]:45969 remote_addr = [::1]:37844 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
14632026-03-10T20:08:06.863ZTRACtest_retry_backoff (dropshot): incoming request local_addr = [::1]:45969 method = PUT remote_addr = [::1]:37844 req_id = a80c2dbc-49cd-4520-a9eb-8222d4cb02dd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
14642026-03-10T20:08:06.863ZDEBGtest_retry_backoff (dropshot): determined request API version local_addr = [::1]:45969 method = PUT remote_addr = [::1]:37844 req_id = a80c2dbc-49cd-4520-a9eb-8222d4cb02dd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config version = 27.0.0
14652026-03-10T20:08:06.867ZINFOtest_retry_backoff (dropshot): request completed latency_us = 4440 local_addr = [::1]:45969 method = PUT remote_addr = [::1]:37844 req_id = a80c2dbc-49cd-4520-a9eb-8222d4cb02dd response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
14662026-03-10T20:08:06.868ZDEBGtest_retry_backoff: client response result = Ok(Response { url: "http://[::1]:45969/omicron-config", status: 204, headers: {"x-request-id": "a80c2dbc-49cd-4520-a9eb-8222d4cb02dd", "date": "Tue, 10 Mar 2026 20:08:06 GMT"} })
14672026-03-10T20:08:06.869ZDEBGtest_retry_backoff: client request body = Some(Body) method = PUT uri = http://[::1]:45969/network-bootstore-config
14682026-03-10T20:08:06.869ZTRACtest_retry_backoff (dropshot): incoming request local_addr = [::1]:45969 method = PUT remote_addr = [::1]:37844 req_id = ececad80-395c-4fe4-8639-8591abd5d034 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
14692026-03-10T20:08:06.869ZDEBGtest_retry_backoff (dropshot): determined request API version local_addr = [::1]:45969 method = PUT remote_addr = [::1]:37844 req_id = ececad80-395c-4fe4-8639-8591abd5d034 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config version = 27.0.0
14702026-03-10T20:08:06.872ZINFOtest_retry_backoff (dropshot): request completed latency_us = 2803 local_addr = [::1]:45969 method = PUT remote_addr = [::1]:37844 req_id = ececad80-395c-4fe4-8639-8591abd5d034 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
14712026-03-10T20:08:06.872ZDEBGtest_retry_backoff: client response result = Ok(Response { url: "http://[::1]:45969/network-bootstore-config", status: 204, headers: {"x-request-id": "ececad80-395c-4fe4-8639-8591abd5d034", "date": "Tue, 10 Mar 2026 20:08:06 GMT"} })
14722026-03-10T20:08:06.872ZDEBGtest_retry_backoff (init_with_steps): Running step start_nexus_external
14732026-03-10T20:08:06.872ZDEBGtest_retry_backoff: Starting Nexus (external API)
14742026-03-10T20:08:06.888ZINFOtest_retry_backoff (BackgroundTasks): deleted 0 of 0 sitreps sitreps background_task = fm_sitrep_gc case_ereports_deleted = 0 cases_deleted = 0 ids = [] sitreps_deleted = 0
14752026-03-10T20:08:06.888ZTRACtest_retry_backoff (BackgroundTasks): all 0 orphaned sitreps in this batch have already been deleted background_task = fm_sitrep_gc
14762026-03-10T20:08:06.888ZINFOtest_retry_backoff (BackgroundTasks): sitrep garbage collection found 0 orphaned sitreps, deleted 0 background_task = fm_sitrep_gc
14772026-03-10T20:08:06.888ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = fm_sitrep_gc elapsed = 1.283457626s iteration = 1 status = {"errors":[],"orphaned_sitreps_deleted":0,"orphaned_sitreps_found":0}
14782026-03-10T20:08:06.888ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Fleet
14792026-03-10T20:08:06.888ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14802026-03-10T20:08:06.889ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database
14812026-03-10T20:08:06.889ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
14822026-03-10T20:08:06.889ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
14832026-03-10T20:08:06.927ZDEBGtest_retry_backoff (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:45969/v2p
14842026-03-10T20:08:06.927ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
14852026-03-10T20:08:06.927ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14862026-03-10T20:08:06.927ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
14872026-03-10T20:08:06.927ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14882026-03-10T20:08:06.928ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14892026-03-10T20:08:06.933ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14902026-03-10T20:08:06.938ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
14912026-03-10T20:08:06.938ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database
14922026-03-10T20:08:06.938ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
14932026-03-10T20:08:06.938ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
14942026-03-10T20:08:06.943ZINFOtest_retry_backoff (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = SimGimlet00
14952026-03-10T20:08:06.943ZDEBGtest_retry_backoff (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:45969/eip-gateways
14962026-03-10T20:08:06.945ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14972026-03-10T20:08:06.945ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
14982026-03-10T20:08:06.945ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14992026-03-10T20:08:06.946ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15002026-03-10T20:08:06.947ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Fleet
15012026-03-10T20:08:06.947ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15022026-03-10T20:08:06.947ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database
15032026-03-10T20:08:06.947ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
15042026-03-10T20:08:06.948ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
15052026-03-10T20:08:06.949ZTRACtest_retry_backoff (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet
15062026-03-10T20:08:06.949ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15072026-03-10T20:08:06.949ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
15082026-03-10T20:08:06.949ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
15092026-03-10T20:08:06.949ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
15102026-03-10T20:08:06.950ZINFOtest_retry_backoff (dropshot): accepted connection local_addr = [::1]:45969 remote_addr = [::1]:46791 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15112026-03-10T20:08:06.950ZTRACtest_retry_backoff (dropshot): incoming request local_addr = [::1]:45969 method = PUT remote_addr = [::1]:60960 req_id = 8353e843-9110-4852-a032-d620a66a2d8a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
15122026-03-10T20:08:06.950ZDEBGtest_retry_backoff (dropshot): determined request API version local_addr = [::1]:45969 method = PUT remote_addr = [::1]:60960 req_id = 8353e843-9110-4852-a032-d620a66a2d8a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways version = 27.0.0
15132026-03-10T20:08:06.951ZINFOtest_retry_backoff (dropshot): request completed latency_us = 670 local_addr = [::1]:45969 method = PUT remote_addr = [::1]:60960 req_id = 8353e843-9110-4852-a032-d620a66a2d8a response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
15142026-03-10T20:08:06.951ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15152026-03-10T20:08:06.951ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
15162026-03-10T20:08:06.951ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15172026-03-10T20:08:06.951ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15182026-03-10T20:08:06.952ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15192026-03-10T20:08:06.956ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
15202026-03-10T20:08:06.956ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database
15212026-03-10T20:08:06.956ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
15222026-03-10T20:08:06.956ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
15232026-03-10T20:08:06.958ZTRACtest_retry_backoff (dropshot): incoming request local_addr = [::1]:45969 method = GET remote_addr = [::1]:46791 req_id = d00e94b6-bc97-4d53-9495-c66dfeea7262 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
15242026-03-10T20:08:06.958ZDEBGtest_retry_backoff (dropshot): determined request API version local_addr = [::1]:45969 method = GET remote_addr = [::1]:46791 req_id = d00e94b6-bc97-4d53-9495-c66dfeea7262 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p version = 27.0.0
15252026-03-10T20:08:06.961ZDEBGtest_retry_backoff (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:45969/eip-gateways", status: 204, headers: {"x-request-id": "8353e843-9110-4852-a032-d620a66a2d8a", "date": "Tue, 10 Mar 2026 20:08:06 GMT"} })
15262026-03-10T20:08:06.961ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 450.54183ms iteration = 2 status = {}
15272026-03-10T20:08:06.962ZINFOtest_retry_backoff (dropshot): request completed latency_us = 4021 local_addr = [::1]:45969 method = GET remote_addr = [::1]:46791 req_id = d00e94b6-bc97-4d53-9495-c66dfeea7262 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
15282026-03-10T20:08:06.962ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15292026-03-10T20:08:06.985ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
15302026-03-10T20:08:06.985ZTRACtest_retry_backoff (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
15312026-03-10T20:08:06.985ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
15322026-03-10T20:08:06.985ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15332026-03-10T20:08:06.985ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15342026-03-10T20:08:06.987ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 1.385407381s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":0,"put_config_err":0,"put_config_ok":0,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":0,"put_config_err":0,"put_config_ok":0,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[]}
15352026-03-10T20:08:06.987ZDEBGtest_retry_backoff (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:45969/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "d00e94b6-bc97-4d53-9495-c66dfeea7262", "content-length": "2", "date": "Tue, 10 Mar 2026 20:08:06 GMT"} })
15362026-03-10T20:08:06.988ZINFOtest_retry_backoff (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = SimGimlet00
15372026-03-10T20:08:06.988ZINFOtest_retry_backoff (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = SimGimlet00
15382026-03-10T20:08:06.988ZINFOtest_retry_backoff (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = SimGimlet00
15392026-03-10T20:08:06.988ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 1.423666946s iteration = 1 status = {}
15402026-03-10T20:08:06.988ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
15412026-03-10T20:08:06.988ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15422026-03-10T20:08:06.988ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15432026-03-10T20:08:06.989ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15442026-03-10T20:08:06.991ZDEBGtest_retry_backoff (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet result = Ok(())
15452026-03-10T20:08:06.991ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
15462026-03-10T20:08:06.991ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
15472026-03-10T20:08:06.992ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
15482026-03-10T20:08:06.993ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
15492026-03-10T20:08:06.993ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15502026-03-10T20:08:06.994ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15512026-03-10T20:08:06.995ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
15522026-03-10T20:08:06.995ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
15532026-03-10T20:08:06.995ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
15542026-03-10T20:08:06.997ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
15552026-03-10T20:08:06.997ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15562026-03-10T20:08:06.998ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15572026-03-10T20:08:07.001ZINFOtest_retry_backoff (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
15582026-03-10T20:08:07.001ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
15592026-03-10T20:08:07.001ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
15602026-03-10T20:08:07.001ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
15612026-03-10T20:08:07.001ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
15622026-03-10T20:08:07.001ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
15632026-03-10T20:08:07.001ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
15642026-03-10T20:08:07.001ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15652026-03-10T20:08:07.002ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15662026-03-10T20:08:07.004ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
15672026-03-10T20:08:07.004ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
15682026-03-10T20:08:07.004ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15692026-03-10T20:08:07.004ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15702026-03-10T20:08:07.008ZTRACtest_retry_backoff (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
15712026-03-10T20:08:07.008ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15722026-03-10T20:08:07.009ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
15732026-03-10T20:08:07.009ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
15742026-03-10T20:08:07.009ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
15752026-03-10T20:08:07.010ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15762026-03-10T20:08:07.010ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
15772026-03-10T20:08:07.010ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15782026-03-10T20:08:07.011ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15792026-03-10T20:08:07.012ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15802026-03-10T20:08:07.042ZDEBGtest_retry_backoff (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
15812026-03-10T20:08:07.042ZDEBGtest_retry_backoff (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
15822026-03-10T20:08:07.042ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
15832026-03-10T20:08:07.042ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
15842026-03-10T20:08:07.042ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
15852026-03-10T20:08:07.053ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15862026-03-10T20:08:07.059ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
15872026-03-10T20:08:07.059ZTRACtest_retry_backoff (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
15882026-03-10T20:08:07.059ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
15892026-03-10T20:08:07.059ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
15902026-03-10T20:08:07.059ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
15912026-03-10T20:08:07.059ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15922026-03-10T20:08:07.060ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15932026-03-10T20:08:07.067ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
15942026-03-10T20:08:07.067ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
15952026-03-10T20:08:07.067ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15962026-03-10T20:08:07.067ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15972026-03-10T20:08:07.068ZINFOtest_retry_backoff (InternalApi): Using static configuration for external switchports
15982026-03-10T20:08:07.069ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
15992026-03-10T20:08:07.069ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
16002026-03-10T20:08:07.069ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
16012026-03-10T20:08:07.070ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16022026-03-10T20:08:07.070ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16032026-03-10T20:08:07.070ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16042026-03-10T20:08:07.071ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16052026-03-10T20:08:07.076ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16062026-03-10T20:08:07.084ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
16072026-03-10T20:08:07.085ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16082026-03-10T20:08:07.085ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16092026-03-10T20:08:07.085ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16102026-03-10T20:08:07.090ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16112026-03-10T20:08:07.090ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16122026-03-10T20:08:07.090ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16132026-03-10T20:08:07.094ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16142026-03-10T20:08:07.094ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16152026-03-10T20:08:07.095ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16162026-03-10T20:08:07.096ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16172026-03-10T20:08:07.096ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16182026-03-10T20:08:07.097ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16192026-03-10T20:08:07.098ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16202026-03-10T20:08:07.098ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16212026-03-10T20:08:07.099ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16222026-03-10T20:08:07.119ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
16232026-03-10T20:08:07.119ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16242026-03-10T20:08:07.120ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16252026-03-10T20:08:07.120ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16262026-03-10T20:08:07.121ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16272026-03-10T20:08:07.126ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16282026-03-10T20:08:07.127ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16292026-03-10T20:08:07.127ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16302026-03-10T20:08:07.127ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16312026-03-10T20:08:07.139ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16322026-03-10T20:08:07.139ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16332026-03-10T20:08:07.139ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16342026-03-10T20:08:07.143ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16352026-03-10T20:08:07.147ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362026-03-10T20:08:07.155ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
16372026-03-10T20:08:07.156ZTRACtest_retry_backoff (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
16382026-03-10T20:08:07.156ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
16392026-03-10T20:08:07.156ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16402026-03-10T20:08:07.156ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16412026-03-10T20:08:07.156ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16422026-03-10T20:08:07.157ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16432026-03-10T20:08:07.158ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
16442026-03-10T20:08:07.158ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
16452026-03-10T20:08:07.158ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
16462026-03-10T20:08:07.161ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16472026-03-10T20:08:07.161ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16482026-03-10T20:08:07.161ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16492026-03-10T20:08:07.162ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16502026-03-10T20:08:07.170ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16512026-03-10T20:08:07.170ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16522026-03-10T20:08:07.170ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16532026-03-10T20:08:07.171ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16542026-03-10T20:08:07.175ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16552026-03-10T20:08:07.181ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
16562026-03-10T20:08:07.181ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16572026-03-10T20:08:07.181ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16582026-03-10T20:08:07.182ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16592026-03-10T20:08:07.199ZDEBGtest_retry_backoff (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
16602026-03-10T20:08:07.199ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Fleet
16612026-03-10T20:08:07.199ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16622026-03-10T20:08:07.199ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16632026-03-10T20:08:07.199ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16642026-03-10T20:08:07.199ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16652026-03-10T20:08:07.209ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16662026-03-10T20:08:07.213ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
16672026-03-10T20:08:07.213ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database
16682026-03-10T20:08:07.213ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16692026-03-10T20:08:07.213ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16702026-03-10T20:08:07.224ZINFOtest_retry_backoff (InternalApi): Recording Rack Network Configuration
16712026-03-10T20:08:07.224ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
16722026-03-10T20:08:07.224ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
16732026-03-10T20:08:07.225ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
16742026-03-10T20:08:07.225ZTRACtest_retry_backoff (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2026-03-10T19:22:50.332390Z, time_modified: 2026-03-10T19:22:50.332390Z, allowed_ips: None } background_task = service_firewall_rule_propagation
16752026-03-10T20:08:07.225ZDEBGtest_retry_backoff (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
16762026-03-10T20:08:07.225ZDEBGtest_retry_backoff (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
16772026-03-10T20:08:07.225ZDEBGtest_retry_backoff (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
16782026-03-10T20:08:07.328ZTRACtest_retry_backoff (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet
16792026-03-10T20:08:07.328ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16802026-03-10T20:08:07.328ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
16812026-03-10T20:08:07.328ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
16822026-03-10T20:08:07.329ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
16832026-03-10T20:08:07.335ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16842026-03-10T20:08:07.338ZDEBGtest_retry_backoff (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet result = Ok(())
16852026-03-10T20:08:07.338ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
16862026-03-10T20:08:07.338ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
16872026-03-10T20:08:07.339ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
16882026-03-10T20:08:07.350ZTRACtest_retry_backoff (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = IpPool { parent: Fleet, key: b00c96f0-f15a-4bcf-b255-f3b64410059b, lookup_type: ByName("oxide-service-pool-v4") }
16892026-03-10T20:08:07.355ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16902026-03-10T20:08:07.355ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
16912026-03-10T20:08:07.355ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
16922026-03-10T20:08:07.355ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
16932026-03-10T20:08:07.360ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16942026-03-10T20:08:07.365ZDEBGtest_retry_backoff (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = IpPool { parent: Fleet, key: b00c96f0-f15a-4bcf-b255-f3b64410059b, lookup_type: ByName("oxide-service-pool-v4") } result = Ok(())
16952026-03-10T20:08:07.365ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
16962026-03-10T20:08:07.365ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
16972026-03-10T20:08:07.365ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
16982026-03-10T20:08:07.381ZTRACtest_retry_backoff (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = IpPool { parent: Fleet, key: 76562722-e11b-4b82-ac71-9c1e25326250, lookup_type: ByName("oxide-service-pool-v6") }
16992026-03-10T20:08:07.381ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17002026-03-10T20:08:07.381ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17012026-03-10T20:08:07.382ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17022026-03-10T20:08:07.382ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17032026-03-10T20:08:07.393ZDEBGtest_retry_backoff (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
17042026-03-10T20:08:07.393ZDEBGtest_retry_backoff (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
17052026-03-10T20:08:07.393ZINFOtest_retry_backoff (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
17062026-03-10T20:08:07.393ZDEBGtest_retry_backoff (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 1.824527038s
17072026-03-10T20:08:07.393ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 1.824553694s iteration = 1 status = {}
17082026-03-10T20:08:07.399ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17092026-03-10T20:08:07.404ZDEBGtest_retry_backoff (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = IpPool { parent: Fleet, key: 76562722-e11b-4b82-ac71-9c1e25326250, lookup_type: ByName("oxide-service-pool-v6") } result = Ok(())
17102026-03-10T20:08:07.404ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17112026-03-10T20:08:07.404ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17122026-03-10T20:08:07.405ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17132026-03-10T20:08:07.421ZTRACtest_retry_backoff (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = IpPool { parent: Fleet, key: b00c96f0-f15a-4bcf-b255-f3b64410059b, lookup_type: ByName("oxide-service-pool-v4") }
17142026-03-10T20:08:07.421ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17152026-03-10T20:08:07.421ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17162026-03-10T20:08:07.421ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17172026-03-10T20:08:07.421ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17182026-03-10T20:08:07.466ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17192026-03-10T20:08:07.470ZDEBGtest_retry_backoff (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = IpPool { parent: Fleet, key: b00c96f0-f15a-4bcf-b255-f3b64410059b, lookup_type: ByName("oxide-service-pool-v4") } result = Ok(())
17202026-03-10T20:08:07.485ZTRACtest_retry_backoff (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = IpPool { parent: Fleet, key: 76562722-e11b-4b82-ac71-9c1e25326250, lookup_type: ByName("oxide-service-pool-v6") }
17212026-03-10T20:08:07.485ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17222026-03-10T20:08:07.485ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17232026-03-10T20:08:07.485ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17242026-03-10T20:08:07.486ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17252026-03-10T20:08:07.496ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262026-03-10T20:08:07.501ZDEBGtest_retry_backoff (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = IpPool { parent: Fleet, key: 76562722-e11b-4b82-ac71-9c1e25326250, lookup_type: ByName("oxide-service-pool-v6") } result = Ok(())
17272026-03-10T20:08:07.520ZTRACtest_retry_backoff (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = IpPool { parent: Fleet, key: b00c96f0-f15a-4bcf-b255-f3b64410059b, lookup_type: ByName("oxide-service-pool-v4") }
17282026-03-10T20:08:07.520ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17292026-03-10T20:08:07.520ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17302026-03-10T20:08:07.520ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17312026-03-10T20:08:07.521ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17322026-03-10T20:08:07.525ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17332026-03-10T20:08:07.528ZDEBGtest_retry_backoff (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = IpPool { parent: Fleet, key: b00c96f0-f15a-4bcf-b255-f3b64410059b, lookup_type: ByName("oxide-service-pool-v4") } result = Ok(())
17342026-03-10T20:08:07.535ZTRACtest_retry_backoff (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = BlueprintConfig
17352026-03-10T20:08:07.535ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17362026-03-10T20:08:07.535ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17372026-03-10T20:08:07.535ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17382026-03-10T20:08:07.535ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17392026-03-10T20:08:07.537ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17402026-03-10T20:08:07.540ZDEBGtest_retry_backoff (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = BlueprintConfig result = Ok(())
17412026-03-10T20:08:07.659ZINFOtest_retry_backoff (InternalApi): inserted blueprint blueprint_id = 6e43be5e-eba9-4a79-84a8-7f9521f2c946
17422026-03-10T20:08:07.659ZTRACtest_retry_backoff (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = BlueprintConfig
17432026-03-10T20:08:07.659ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17442026-03-10T20:08:07.659ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17452026-03-10T20:08:07.659ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17462026-03-10T20:08:07.660ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17472026-03-10T20:08:07.662ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17482026-03-10T20:08:07.665ZDEBGtest_retry_backoff (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = BlueprintConfig result = Ok(())
17492026-03-10T20:08:07.681ZINFOtest_retry_backoff (InternalApi): No networking records needed for crucible_pantry service
17502026-03-10T20:08:07.681ZINFOtest_retry_backoff (InternalApi): No networking records needed for clickhouse service
17512026-03-10T20:08:07.728ZINFOtest_retry_backoff (InternalApi): Inserted networking records for boundary_ntp service
17522026-03-10T20:08:07.746ZINFOtest_retry_backoff (InternalApi): Inserted networking records for external_dns service
17532026-03-10T20:08:07.746ZINFOtest_retry_backoff (InternalApi): No networking records needed for internal_dns service
17542026-03-10T20:08:07.765ZINFOtest_retry_backoff (InternalApi): Inserted networking records for nexus service
17552026-03-10T20:08:07.765ZINFOtest_retry_backoff (InternalApi): No networking records needed for cockroach_db service
17562026-03-10T20:08:07.765ZINFOtest_retry_backoff (InternalApi): Inserted service networking records
17572026-03-10T20:08:07.765ZINFOtest_retry_backoff (InternalApi): Inserted physical disks
17582026-03-10T20:08:07.765ZINFOtest_retry_backoff (InternalApi): Inserted zpools
17592026-03-10T20:08:07.765ZINFOtest_retry_backoff (InternalApi): Inserted datasets
17602026-03-10T20:08:07.772ZINFOtest_retry_backoff (InternalApi): Populated DNS tables for internal DNS
17612026-03-10T20:08:07.774ZINFOtest_retry_backoff (InternalApi): Populated DNS tables for external DNS
17622026-03-10T20:08:07.774ZTRACtest_retry_backoff (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet
17632026-03-10T20:08:07.774ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17642026-03-10T20:08:07.774ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17652026-03-10T20:08:07.774ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17662026-03-10T20:08:07.775ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17672026-03-10T20:08:07.777ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17682026-03-10T20:08:07.780ZDEBGtest_retry_backoff (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet result = Ok(())
17692026-03-10T20:08:07.780ZTRACtest_retry_backoff (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet
17702026-03-10T20:08:07.780ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17712026-03-10T20:08:07.780ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17722026-03-10T20:08:07.780ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17732026-03-10T20:08:07.781ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17742026-03-10T20:08:07.783ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17752026-03-10T20:08:07.785ZDEBGtest_retry_backoff (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet result = Ok(())
17762026-03-10T20:08:07.805ZTRACtest_retry_backoff (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = DnsConfig
17772026-03-10T20:08:07.805ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17782026-03-10T20:08:07.805ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17792026-03-10T20:08:07.805ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17802026-03-10T20:08:07.805ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17812026-03-10T20:08:07.808ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17822026-03-10T20:08:07.812ZDEBGtest_retry_backoff (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = DnsConfig result = Ok(())
17832026-03-10T20:08:07.812ZTRACtest_retry_backoff (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = DnsConfig
17842026-03-10T20:08:07.812ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17852026-03-10T20:08:07.812ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17862026-03-10T20:08:07.812ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17872026-03-10T20:08:07.813ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17882026-03-10T20:08:07.815ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17892026-03-10T20:08:07.820ZDEBGtest_retry_backoff (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = DnsConfig result = Ok(())
17902026-03-10T20:08:07.821ZTRACtest_retry_backoff (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = DnsConfig
17912026-03-10T20:08:07.821ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17922026-03-10T20:08:07.821ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
17932026-03-10T20:08:07.821ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
17942026-03-10T20:08:07.822ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
17952026-03-10T20:08:07.827ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17962026-03-10T20:08:07.832ZDEBGtest_retry_backoff (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = DnsConfig result = Ok(())
17972026-03-10T20:08:07.837ZINFOtest_retry_backoff (InternalApi): Created recovery silo
17982026-03-10T20:08:07.837ZINFOtest_retry_backoff (InternalApi): Created recovery user
17992026-03-10T20:08:07.840ZINFOtest_retry_backoff (InternalApi): Created recovery user's password
18002026-03-10T20:08:07.840ZTRACtest_retry_backoff (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Silo { parent: Fleet, key: d190bccd-cbb4-4881-8f1a-85f146780c84, lookup_type: ById(d190bccd-cbb4-4881-8f1a-85f146780c84) }
18012026-03-10T20:08:07.840ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = d190bccd-cbb4-4881-8f1a-85f146780c84 resource_type = Silo
18022026-03-10T20:08:07.840ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
18032026-03-10T20:08:07.840ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
18042026-03-10T20:08:07.840ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
18052026-03-10T20:08:07.842ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18062026-03-10T20:08:07.842ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
18072026-03-10T20:08:07.842ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
18082026-03-10T20:08:07.842ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
18092026-03-10T20:08:07.844ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18102026-03-10T20:08:07.846ZDEBGtest_retry_backoff (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Silo { parent: Fleet, key: d190bccd-cbb4-4881-8f1a-85f146780c84, lookup_type: ById(d190bccd-cbb4-4881-8f1a-85f146780c84) } result = Ok(())
18112026-03-10T20:08:07.846ZDEBGtest_retry_backoff (InternalApi): Generated role assignment queries
18122026-03-10T20:08:07.847ZINFOtest_retry_backoff (InternalApi): Granted Silo privileges
18132026-03-10T20:08:07.847ZTRACtest_retry_backoff (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet
18142026-03-10T20:08:07.847ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18152026-03-10T20:08:07.847ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
18162026-03-10T20:08:07.847ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
18172026-03-10T20:08:07.848ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
18182026-03-10T20:08:07.851ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18192026-03-10T20:08:07.852ZDEBGtest_retry_backoff (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet result = Ok(())
18202026-03-10T20:08:07.856ZTRACtest_retry_backoff (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
18212026-03-10T20:08:07.856ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18222026-03-10T20:08:07.856ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
18232026-03-10T20:08:07.856ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
18242026-03-10T20:08:07.857ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
18252026-03-10T20:08:07.859ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18262026-03-10T20:08:07.862ZDEBGtest_retry_backoff (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
18272026-03-10T20:08:07.862ZTRACtest_retry_backoff (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
18282026-03-10T20:08:07.862ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18292026-03-10T20:08:07.862ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
18302026-03-10T20:08:07.862ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
18312026-03-10T20:08:07.862ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
18322026-03-10T20:08:07.864ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18332026-03-10T20:08:07.867ZDEBGtest_retry_backoff (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
18342026-03-10T20:08:07.887ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18352026-03-10T20:08:07.887ZDEBGtest_retry_backoff (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18362026-03-10T20:08:07.887ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database
18372026-03-10T20:08:07.887ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18382026-03-10T20:08:07.888ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18392026-03-10T20:08:07.888ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18402026-03-10T20:08:07.888ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18412026-03-10T20:08:07.888ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18422026-03-10T20:08:07.888ZDEBGtest_retry_backoff (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18432026-03-10T20:08:07.888ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database
18442026-03-10T20:08:07.889ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18452026-03-10T20:08:07.889ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18462026-03-10T20:08:07.889ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18472026-03-10T20:08:07.889ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18482026-03-10T20:08:07.889ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18492026-03-10T20:08:07.889ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Fleet
18502026-03-10T20:08:07.889ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18512026-03-10T20:08:07.889ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database
18522026-03-10T20:08:07.889ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18532026-03-10T20:08:07.890ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database result = Ok(())
18542026-03-10T20:08:07.890ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18552026-03-10T20:08:07.890ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Inventory
18562026-03-10T20:08:07.890ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18572026-03-10T20:08:07.890ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
18582026-03-10T20:08:07.890ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18592026-03-10T20:08:07.891ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
18602026-03-10T20:08:07.891ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = blueprint_loader iteration = 2 reason = Signaled
18612026-03-10T20:08:07.891ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = BlueprintConfig
18622026-03-10T20:08:07.891ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18632026-03-10T20:08:07.891ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database
18642026-03-10T20:08:07.891ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18652026-03-10T20:08:07.891ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database result = Ok(())
18662026-03-10T20:08:07.892ZTRACtest_retry_backoff (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: a1 b6 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:23005 req_id = 47ded123-7dcf-4aed-b43c-0f57e51f47b9
18672026-03-10T20:08:07.892ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41398,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:23005 req_id = 47ded123-7dcf-4aed-b43c-0f57e51f47b9
18682026-03-10T20:08:07.893ZDEBGtest_retry_backoff (store): query key key = _nameservice._tcp
18692026-03-10T20:08:07.893ZDEBGtest_retry_backoff (store): query key key = 81796aa6-4ecc-4d56-a811-dc112da8da6c.host
18702026-03-10T20:08:07.893ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("81796aa6-4ecc-4d56-a811-dc112da8da6c.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:23005 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 64860, target: Name("81796aa6-4ecc-4d56-a811-dc112da8da6c.host.control-plane.oxide.internal") }) }] req_id = 47ded123-7dcf-4aed-b43c-0f57e51f47b9
18712026-03-10T20:08:07.893ZTRACtest_retry_backoff (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 92 61 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 .a..........._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52369 req_id = 3b89bd7a-f92c-45da-9c6d-0abc815c029a
18722026-03-10T20:08:07.893ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37473,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:52369 req_id = 3b89bd7a-f92c-45da-9c6d-0abc815c029a
18732026-03-10T20:08:07.893ZDEBGtest_retry_backoff (store): query key key = _external-dns._tcp
18742026-03-10T20:08:07.893ZDEBGtest_retry_backoff (store): query key key = 61d3324b-3a36-494c-95ab-02b65214832c.host
18752026-03-10T20:08:07.893ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("61d3324b-3a36-494c-95ab-02b65214832c.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:52369 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 61901, target: Name("61d3324b-3a36-494c-95ab-02b65214832c.host.control-plane.oxide.internal") }) }] req_id = 3b89bd7a-f92c-45da-9c6d-0abc815c029a
18762026-03-10T20:08:07.894ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 64860, target: Name("81796aa6-4ecc-4d56-a811-dc112da8da6c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("81796aa6-4ecc-4d56-a811-dc112da8da6c.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6307, tv_nsec: 753899585 } })
18772026-03-10T20:08:07.894ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
18782026-03-10T20:08:07.894ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 61901, target: Name("61d3324b-3a36-494c-95ab-02b65214832c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("61d3324b-3a36-494c-95ab-02b65214832c.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6307, tv_nsec: 754181257 } })
18792026-03-10T20:08:07.894ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
18802026-03-10T20:08:07.894ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
18812026-03-10T20:08:07.894ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18822026-03-10T20:08:07.894ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database
18832026-03-10T20:08:07.894ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18842026-03-10T20:08:07.894ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18852026-03-10T20:08:07.895ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
18862026-03-10T20:08:07.895ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872026-03-10T20:08:07.895ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database
18882026-03-10T20:08:07.895ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18892026-03-10T20:08:07.895ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18902026-03-10T20:08:07.896ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18912026-03-10T20:08:07.900ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18922026-03-10T20:08:07.900ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database
18932026-03-10T20:08:07.900ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18942026-03-10T20:08:07.900ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database result = Ok(())
18952026-03-10T20:08:07.901ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18962026-03-10T20:08:07.905ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Fleet result = Ok(())
18972026-03-10T20:08:07.905ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database
18982026-03-10T20:08:07.905ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18992026-03-10T20:08:07.905ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database result = Ok(())
19002026-03-10T20:08:07.906ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19012026-03-10T20:08:07.910ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19022026-03-10T20:08:07.910ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
19032026-03-10T20:08:07.910ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19042026-03-10T20:08:07.910ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
19052026-03-10T20:08:07.911ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19062026-03-10T20:08:07.916ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19072026-03-10T20:08:07.916ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19082026-03-10T20:08:07.923ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19092026-03-10T20:08:07.930ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6e43be5e-eba9-4a79-84a8-7f9521f2c946, lookup_type: ById(6e43be5e-eba9-4a79-84a8-7f9521f2c946) }
19102026-03-10T20:08:07.930ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19112026-03-10T20:08:07.930ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database
19122026-03-10T20:08:07.930ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19132026-03-10T20:08:07.931ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database result = Ok(())
19142026-03-10T20:08:07.932ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = DnsConfig
19152026-03-10T20:08:07.932ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19162026-03-10T20:08:07.932ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database
19172026-03-10T20:08:07.932ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19182026-03-10T20:08:07.932ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database result = Ok(())
19192026-03-10T20:08:07.934ZDEBGtest_retry_backoff (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19202026-03-10T20:08:07.934ZDEBGtest_retry_backoff (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19212026-03-10T20:08:07.934ZDEBGtest_retry_backoff (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19222026-03-10T20:08:07.934ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19232026-03-10T20:08:07.934ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19242026-03-10T20:08:07.934ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database
19252026-03-10T20:08:07.934ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19262026-03-10T20:08:07.934ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19272026-03-10T20:08:07.936ZDEBGtest_retry_backoff (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
19282026-03-10T20:08:07.936ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19292026-03-10T20:08:07.938ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19302026-03-10T20:08:07.943ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6e43be5e-eba9-4a79-84a8-7f9521f2c946, lookup_type: ById(6e43be5e-eba9-4a79-84a8-7f9521f2c946) } result = Ok(())
19312026-03-10T20:08:07.943ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database
19322026-03-10T20:08:07.943ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19332026-03-10T20:08:07.943ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_loader resource = Database result = Ok(())
19342026-03-10T20:08:07.944ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19352026-03-10T20:08:07.949ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19362026-03-10T20:08:07.949ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database
19372026-03-10T20:08:07.949ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19382026-03-10T20:08:07.949ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database result = Ok(())
19392026-03-10T20:08:07.950ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19402026-03-10T20:08:07.957ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19412026-03-10T20:08:07.957ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database
19422026-03-10T20:08:07.957ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19432026-03-10T20:08:07.957ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19442026-03-10T20:08:07.960ZTRACtest_retry_backoff (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: bf e3 01 00 00 01 00 00 00 00 00 01 24 38 31 37 ............$817\n0010: 39 36 61 61 36 2d 34 65 63 63 2d 34 64 35 36 2d 96aa6-4ecc-4d56-\n0020: 61 38 31 31 2d 64 63 31 31 32 64 61 38 64 61 36 a811-dc112da8da6\n0030: 63 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 c.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:40349 req_id = 3525d299-9bd9-4a9e-852b-4bb1490f0fd0
19452026-03-10T20:08:07.961ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49123,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("81796aa6-4ecc-4d56-a811-dc112da8da6c.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("81796aa6-4ecc-4d56-a811-dc112da8da6c.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 36,\n 56,\n 49,\n 55,\n 57,\n 54,\n 97,\n 97,\n 54,\n 45,\n 52,\n 101,\n 99,\n 99,\n 45,\n 52,\n 100,\n 53,\n 54,\n 45,\n 97,\n 56,\n 49,\n 49,\n 45,\n 100,\n 99,\n 49,\n 49,\n 50,\n 100,\n 97,\n 56,\n 100,\n 97,\n 54,\n 99,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:40349 req_id = 3525d299-9bd9-4a9e-852b-4bb1490f0fd0
19462026-03-10T20:08:07.961ZDEBGtest_retry_backoff (store): query key key = 81796aa6-4ecc-4d56-a811-dc112da8da6c.host
19472026-03-10T20:08:07.961ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:40349 query = LowerQuery { name: LowerName(Name("81796aa6-4ecc-4d56-a811-dc112da8da6c.host.control-plane.oxide.internal.")), original: Query { name: Name("81796aa6-4ecc-4d56-a811-dc112da8da6c.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("81796aa6-4ecc-4d56-a811-dc112da8da6c.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 3525d299-9bd9-4a9e-852b-4bb1490f0fd0
19482026-03-10T20:08:07.961ZTRACtest_retry_backoff (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: f8 86 01 00 00 01 00 00 00 00 00 01 24 36 31 64 ............$61d\n0010: 33 33 32 34 62 2d 33 61 33 36 2d 34 39 34 63 2d 3324b-3a36-494c-\n0020: 39 35 61 62 2d 30 32 62 36 35 32 31 34 38 33 32 95ab-02b65214832\n0030: 63 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 c.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:11636 req_id = 39df201a-fae1-4086-b4c2-e31f17102180
19492026-03-10T20:08:07.961ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63622,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("61d3324b-3a36-494c-95ab-02b65214832c.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("61d3324b-3a36-494c-95ab-02b65214832c.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 36,\n 54,\n 49,\n 100,\n 51,\n 51,\n 50,\n 52,\n 98,\n 45,\n 51,\n 97,\n 51,\n 54,\n 45,\n 52,\n 57,\n 52,\n 99,\n 45,\n 57,\n 53,\n 97,\n 98,\n 45,\n 48,\n 50,\n 98,\n 54,\n 53,\n 50,\n 49,\n 52,\n 56,\n 51,\n 50,\n 99,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:11636 req_id = 39df201a-fae1-4086-b4c2-e31f17102180
19502026-03-10T20:08:07.961ZDEBGtest_retry_backoff (store): query key key = 61d3324b-3a36-494c-95ab-02b65214832c.host
19512026-03-10T20:08:07.961ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:11636 query = LowerQuery { name: LowerName(Name("61d3324b-3a36-494c-95ab-02b65214832c.host.control-plane.oxide.internal.")), original: Query { name: Name("61d3324b-3a36-494c-95ab-02b65214832c.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("61d3324b-3a36-494c-95ab-02b65214832c.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 39df201a-fae1-4086-b4c2-e31f17102180
19522026-03-10T20:08:07.962ZINFOtest_retry_backoff (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:64860] } background_task = dns_servers_internal dns_group = internal
19532026-03-10T20:08:07.962ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 73.603406ms iteration = 2 status = {"addresses":["[::1]:64860"]}
19542026-03-10T20:08:07.962ZINFOtest_retry_backoff (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:61901] } background_task = dns_servers_external dns_group = external
19552026-03-10T20:08:07.962ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 72.805251ms iteration = 2 status = {"addresses":["[::1]:61901"]}
19562026-03-10T20:08:07.962ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b2 56 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .V..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:23774 req_id = e8849416-9b3e-4ecb-aa9f-5fb8fe036378
19572026-03-10T20:08:07.962ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45654,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:23774 req_id = e8849416-9b3e-4ecb-aa9f-5fb8fe036378
19582026-03-10T20:08:07.964ZDEBGtest_retry_backoff (store): query key key = _mgs._tcp
19592026-03-10T20:08:07.964ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19602026-03-10T20:08:07.964ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:23774 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 43524, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = e8849416-9b3e-4ecb-aa9f-5fb8fe036378
19612026-03-10T20:08:07.964ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19622026-03-10T20:08:07.964ZWARNtest_retry_backoff (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19632026-03-10T20:08:07.964ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 156.35\u{b5}s iteration = 2 status = {"error":"no config"}
19642026-03-10T20:08:07.964ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19652026-03-10T20:08:07.964ZWARNtest_retry_backoff (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19662026-03-10T20:08:07.964ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 6.522\u{b5}s iteration = 2 status = {"error":"no config"}
19672026-03-10T20:08:07.965ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Fleet
19682026-03-10T20:08:07.965ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19692026-03-10T20:08:07.965ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database
19702026-03-10T20:08:07.965ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19712026-03-10T20:08:07.967ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database result = Ok(())
19722026-03-10T20:08:07.969ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19732026-03-10T20:08:07.980ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19742026-03-10T20:08:07.980ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database
19752026-03-10T20:08:07.980ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19762026-03-10T20:08:07.980ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = external_endpoints resource = Database result = Ok(())
19772026-03-10T20:08:07.982ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 43524, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6307, tv_nsec: 842766167 } })
19782026-03-10T20:08:07.983ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
19792026-03-10T20:08:07.983ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 99 c3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54963 req_id = 2b47ef04-462b-46f8-9629-007861a29554
19802026-03-10T20:08:07.983ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39363,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:54963 req_id = 2b47ef04-462b-46f8-9629-007861a29554
19812026-03-10T20:08:07.983ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19822026-03-10T20:08:07.984ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54963 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 2b47ef04-462b-46f8-9629-007861a29554
19832026-03-10T20:08:08.027ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
19842026-03-10T20:08:08.028ZDEBGtest_retry_backoff (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19852026-03-10T20:08:08.028ZDEBGtest_retry_backoff (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19862026-03-10T20:08:08.028ZDEBGtest_retry_backoff (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19872026-03-10T20:08:08.028ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19882026-03-10T20:08:08.028ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19892026-03-10T20:08:08.028ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database
19902026-03-10T20:08:08.028ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19912026-03-10T20:08:08.028ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19922026-03-10T20:08:08.029ZDEBGtest_retry_backoff (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
19932026-03-10T20:08:08.029ZDEBGtest_retry_backoff (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 10cee23c-a7a4-4d84-a2e9-7192291ba01a dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
19942026-03-10T20:08:08.029ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19952026-03-10T20:08:08.029ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19962026-03-10T20:08:08.029ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database
19972026-03-10T20:08:08.029ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19982026-03-10T20:08:08.030ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19992026-03-10T20:08:08.031ZINFOtest_retry_backoff (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2026-03-10T20:08:04.676756Z, time_modified: 2026-03-10T20:08:04.676756Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)), admin_group_name: None }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: d190bccd-cbb4-4881-8f1a-85f146780c84, db_silo: Silo { identity: SiloIdentity { id: d190bccd-cbb4-4881-8f1a-85f146780c84, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2026-03-10T20:08:07.774717Z, time_modified: 2026-03-10T20:08:07.774717Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)), admin_group_name: None }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: d190bccd-cbb4-4881-8f1a-85f146780c84, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: d190bccd-cbb4-4881-8f1a-85f146780c84, db_silo: Silo { identity: SiloIdentity { id: d190bccd-cbb4-4881-8f1a-85f146780c84, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2026-03-10T20:08:07.774717Z, time_modified: 2026-03-10T20:08:07.774717Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)), admin_group_name: None }, tls_certs: [] }) }
20002026-03-10T20:08:08.031ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 141.362577ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"d190bccd-cbb4-4881-8f1a-85f146780c84","tls_certs":[]}},"default_endpoint":{"silo_id":"d190bccd-cbb4-4881-8f1a-85f146780c84","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo d190bccd-cbb4-4881-8f1a-85f146780c84 with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
20012026-03-10T20:08:08.032ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20022026-03-10T20:08:08.039ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20032026-03-10T20:08:08.040ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database
20042026-03-10T20:08:08.040ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20052026-03-10T20:08:08.040ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20062026-03-10T20:08:08.041ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20072026-03-10T20:08:08.045ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20082026-03-10T20:08:08.045ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database
20092026-03-10T20:08:08.045ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20102026-03-10T20:08:08.046ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20112026-03-10T20:08:08.047ZTRACtest_retry_backoff (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b3 fe 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60943 req_id = 4dc4678d-abe6-4f41-ae8e-598e8ce85128
20122026-03-10T20:08:08.047ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46078,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:60943 req_id = 4dc4678d-abe6-4f41-ae8e-598e8ce85128
20132026-03-10T20:08:08.047ZDEBGtest_retry_backoff (store): query key key = _clickhouse-admin-keeper._tcp
20142026-03-10T20:08:08.047ZERROtest_retry_backoff (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46078, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal kind = dns peer_addr = [::1]:60943 req_id = 4dc4678d-abe6-4f41-ae8e-598e8ce85128
20152026-03-10T20:08:08.047ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _cockroach._tcp.control-plane.oxide.internal
20162026-03-10T20:08:08.048ZDEBGtest_retry_backoff (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20172026-03-10T20:08:08.048ZDEBGtest_retry_backoff (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 6d4eb468-0d30-4ff6-baca-6c7b0368dfb5 dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
20182026-03-10T20:08:08.048ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
20192026-03-10T20:08:08.048ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20202026-03-10T20:08:08.048ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database
20212026-03-10T20:08:08.048ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20222026-03-10T20:08:08.048ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20232026-03-10T20:08:08.049ZDEBGtest_retry_backoff (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 10cee23c-a7a4-4d84-a2e9-7192291ba01a dns_zone_name = control-plane.oxide.internal found_so_far = 27 version = 1
20242026-03-10T20:08:08.049ZDEBGtest_retry_backoff (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20252026-03-10T20:08:08.049ZINFOtest_retry_backoff (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20262026-03-10T20:08:08.049ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 161.965469ms iteration = 2 status = {"generation":1}
20272026-03-10T20:08:08.049ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20282026-03-10T20:08:08.091ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:64860 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:64860] } uri = http://[::1]:64860/config
20292026-03-10T20:08:08.092ZINFOtest_retry_backoff (http): accepted connection kind = dns local_addr = [::1]:64860 remote_addr = [::1]:33176
20302026-03-10T20:08:08.093ZTRACtest_retry_backoff (http): incoming request kind = dns local_addr = [::1]:64860 method = PUT remote_addr = [::1]:33176 req_id = 7a6caf8c-0398-4fe9-ace7-70ddb93b76df uri = /config
20312026-03-10T20:08:08.093ZDEBGtest_retry_backoff (http): determined request API version kind = dns local_addr = [::1]:64860 method = PUT remote_addr = [::1]:33176 req_id = 7a6caf8c-0398-4fe9-ace7-70ddb93b76df uri = /config version = 2.0.0
20322026-03-10T20:08:08.093ZTRACtest_retry_backoff (dns): buffer buffer = Length: 73 (0x49) bytes\n0000: b4 25 01 00 00 01 00 00 00 00 00 01 0a 5f 63 6f .%..........._co\n0010: 63 6b 72 6f 61 63 68 04 5f 74 63 70 0d 63 6f 6e ckroach._tcp.con\n0020: 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 trol-plane.oxide\n0030: 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 .internal..!....\n0040: 29 04 d0 00 00 00 00 00 00 )........ kind = dns peer_addr = [::1]:17218 req_id = dbe4e6c3-93a8-4521-8469-a947ceba00b6
20332026-03-10T20:08:08.093ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46117,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_cockroach._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_cockroach._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 10,\n 95,\n 99,\n 111,\n 99,\n 107,\n 114,\n 111,\n 97,\n 99,\n 104,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:17218 req_id = dbe4e6c3-93a8-4521-8469-a947ceba00b6
20342026-03-10T20:08:08.093ZDEBGtest_retry_backoff (store): query key key = _cockroach._tcp
20352026-03-10T20:08:08.093ZDEBGtest_retry_backoff (store): query key key = f189113c-85eb-4d6a-b2fc-46ba3de160e6.host
20362026-03-10T20:08:08.093ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("f189113c-85eb-4d6a-b2fc-46ba3de160e6.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:17218 query = LowerQuery { name: LowerName(Name("_cockroach._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_cockroach._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_cockroach._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 42738, target: Name("f189113c-85eb-4d6a-b2fc-46ba3de160e6.host.control-plane.oxide.internal") }) }] req_id = dbe4e6c3-93a8-4521-8469-a947ceba00b6
20372026-03-10T20:08:08.093ZINFOtest_retry_backoff (store): attempting generation update new_generation = 1 req_id = 7a6caf8c-0398-4fe9-ace7-70ddb93b76df
20382026-03-10T20:08:08.093ZINFOtest_retry_backoff (store): updated generation new_generation = 1 req_id = 7a6caf8c-0398-4fe9-ace7-70ddb93b76df
20392026-03-10T20:08:08.093ZINFOtest_retry_backoff (http): request completed kind = dns latency_us = 953 local_addr = [::1]:64860 method = PUT remote_addr = [::1]:33176 req_id = 7a6caf8c-0398-4fe9-ace7-70ddb93b76df response_code = 204 uri = /config
20402026-03-10T20:08:08.094ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20412026-03-10T20:08:08.098ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20422026-03-10T20:08:08.098ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database
20432026-03-10T20:08:08.099ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20442026-03-10T20:08:08.099ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20452026-03-10T20:08:08.100ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _cockroach._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_cockroach._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_cockroach._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 42738, target: Name("f189113c-85eb-4d6a-b2fc-46ba3de160e6.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("f189113c-85eb-4d6a-b2fc-46ba3de160e6.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6307, tv_nsec: 960326777 } })
20462026-03-10T20:08:08.100ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
20472026-03-10T20:08:08.100ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:64860 generation = 1 result = Ok(Response { url: "http://[::1]:64860/config", status: 204, headers: {"x-request-id": "7a6caf8c-0398-4fe9-ace7-70ddb93b76df", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} }) servers = DnsServersList { addresses: [[::1]:64860] }
20482026-03-10T20:08:08.100ZINFOtest_retry_backoff (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:64860 generation = 1 servers = DnsServersList { addresses: [[::1]:64860] }
20492026-03-10T20:08:08.100ZINFOtest_retry_backoff (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:64860] }
20502026-03-10T20:08:08.100ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 51.442553ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:64860":{"Ok":null}}}
20512026-03-10T20:08:08.101ZTRACtest_retry_backoff (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 29 53 01 00 00 01 00 00 00 00 00 01 24 66 31 38 )S..........$f18\n0010: 39 31 31 33 63 2d 38 35 65 62 2d 34 64 36 61 2d 9113c-85eb-4d6a-\n0020: 62 32 66 63 2d 34 36 62 61 33 64 65 31 36 30 65 b2fc-46ba3de160e\n0030: 36 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 6.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:18740 req_id = cc96d931-e917-4c08-a359-71a0f42a9a89
20522026-03-10T20:08:08.101ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10579,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("f189113c-85eb-4d6a-b2fc-46ba3de160e6.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("f189113c-85eb-4d6a-b2fc-46ba3de160e6.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 36,\n 102,\n 49,\n 56,\n 57,\n 49,\n 49,\n 51,\n 99,\n 45,\n 56,\n 53,\n 101,\n 98,\n 45,\n 52,\n 100,\n 54,\n 97,\n 45,\n 98,\n 50,\n 102,\n 99,\n 45,\n 52,\n 54,\n 98,\n 97,\n 51,\n 100,\n 101,\n 49,\n 54,\n 48,\n 101,\n 54,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:18740 req_id = cc96d931-e917-4c08-a359-71a0f42a9a89
20532026-03-10T20:08:08.101ZDEBGtest_retry_backoff (store): query key key = f189113c-85eb-4d6a-b2fc-46ba3de160e6.host
20542026-03-10T20:08:08.101ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:18740 query = LowerQuery { name: LowerName(Name("f189113c-85eb-4d6a-b2fc-46ba3de160e6.host.control-plane.oxide.internal.")), original: Query { name: Name("f189113c-85eb-4d6a-b2fc-46ba3de160e6.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("f189113c-85eb-4d6a-b2fc-46ba3de160e6.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = cc96d931-e917-4c08-a359-71a0f42a9a89
20552026-03-10T20:08:08.140ZDEBGtest_retry_backoff (CockroachClusterAdminClient): Updated CockroachDB cluster backends added = 1 addresses = [[::1]:32222] removed = 0 total = 1
20562026-03-10T20:08:08.140ZDEBGtest_retry_backoff (BackgroundTasks): begin collection background_task = inventory_collection
20572026-03-10T20:08:08.140ZDEBGtest_retry_backoff (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:43524
20582026-03-10T20:08:08.140ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/ignition
20592026-03-10T20:08:08.140ZDEBGtest_retry_backoff (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 6d4eb468-0d30-4ff6-baca-6c7b0368dfb5 dns_zone_name = oxide-dev.test found_so_far = 3 version = 2
20602026-03-10T20:08:08.140ZDEBGtest_retry_backoff (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20612026-03-10T20:08:08.140ZINFOtest_retry_backoff (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20622026-03-10T20:08:08.140ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 252.047724ms iteration = 2 status = {"generation":2}
20632026-03-10T20:08:08.141ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20642026-03-10T20:08:08.178ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:61901 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:61901] } uri = http://[::1]:61901/config
20652026-03-10T20:08:08.178ZINFOtest_retry_backoff (http): accepted connection kind = dns local_addr = [::1]:61901 remote_addr = [::1]:45225
20662026-03-10T20:08:08.179ZTRACtest_retry_backoff (http): incoming request kind = dns local_addr = [::1]:61901 method = PUT remote_addr = [::1]:45225 req_id = 615ebb9b-5299-43dc-b08f-c264a472a769 uri = /config
20672026-03-10T20:08:08.179ZDEBGtest_retry_backoff (http): determined request API version kind = dns local_addr = [::1]:61901 method = PUT remote_addr = [::1]:45225 req_id = 615ebb9b-5299-43dc-b08f-c264a472a769 uri = /config version = 2.0.0
20682026-03-10T20:08:08.179ZINFOtest_retry_backoff (store): attempting generation update new_generation = 2 req_id = 615ebb9b-5299-43dc-b08f-c264a472a769
20692026-03-10T20:08:08.179ZINFOtest_retry_backoff (store): pruning trees for generations newer than 0
20702026-03-10T20:08:08.179ZDEBGtest_retry_backoff (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20712026-03-10T20:08:08.179ZDEBGtest_retry_backoff (store): updating current config
20722026-03-10T20:08:08.179ZDEBGtest_retry_backoff (store): flushing default tree
20732026-03-10T20:08:08.179ZINFOtest_retry_backoff (store): pruning trees for generations older than 2 keep = 3
20742026-03-10T20:08:08.179ZINFOtest_retry_backoff (store): updated generation new_generation = 2 req_id = 615ebb9b-5299-43dc-b08f-c264a472a769
20752026-03-10T20:08:08.179ZINFOtest_retry_backoff (http): request completed kind = dns latency_us = 760 local_addr = [::1]:61901 method = PUT remote_addr = [::1]:45225 req_id = 615ebb9b-5299-43dc-b08f-c264a472a769 response_code = 204 uri = /config
20762026-03-10T20:08:08.180ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:61901 generation = 2 result = Ok(Response { url: "http://[::1]:61901/config", status: 204, headers: {"x-request-id": "615ebb9b-5299-43dc-b08f-c264a472a769", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} }) servers = DnsServersList { addresses: [[::1]:61901] }
20772026-03-10T20:08:08.180ZINFOtest_retry_backoff (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:61901 generation = 2 servers = DnsServersList { addresses: [[::1]:61901] }
20782026-03-10T20:08:08.180ZINFOtest_retry_backoff (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:61901] }
20792026-03-10T20:08:08.180ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 39.390696ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:61901":{"Ok":null}}}
20802026-03-10T20:08:08.181ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a154fa8-b0d1-4601-9f8a-55697f35c7c4", "content-length": "835", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
20812026-03-10T20:08:08.181ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0
20822026-03-10T20:08:08.182ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "602345b4-64b2-4077-b513-0c9fb7b99959", "content-length": "734", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
20832026-03-10T20:08:08.183ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0/component/sp/caboose?firmware_slot=0
20842026-03-10T20:08:08.184ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15b47908-560c-4f8e-a06f-898b5f7a934c", "content-length": "111", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
20852026-03-10T20:08:08.184ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0/component/sp/caboose?firmware_slot=1
20862026-03-10T20:08:08.185ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63ac8b64-0b56-4dd6-b73e-ce0439139dc3", "content-length": "111", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
20872026-03-10T20:08:08.185ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0/component/rot/caboose?firmware_slot=0
20882026-03-10T20:08:08.186ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4a76ce2-93e5-4e16-b544-268d0f61aa21", "content-length": "163", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
20892026-03-10T20:08:08.186ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0/component/rot/caboose?firmware_slot=1
20902026-03-10T20:08:08.187ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b06de22f-d58e-4e52-999b-87c47a7a2b57", "content-length": "163", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
20912026-03-10T20:08:08.187ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0/component/stage0/caboose?firmware_slot=0
20922026-03-10T20:08:08.188ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4248f722-7861-4c76-87e0-e0025383ee1a", "content-length": "166", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
20932026-03-10T20:08:08.188ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0/component/stage0/caboose?firmware_slot=1
20942026-03-10T20:08:08.189ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33637dd6-0b0c-4e9e-97ca-b13900637f54", "content-length": "166", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
20952026-03-10T20:08:08.189ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0/component/rot/cmpa
20962026-03-10T20:08:08.189ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af71b885-1c7b-4378-ba35-250b112ed1d9", "content-length": "702", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
20972026-03-10T20:08:08.190ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0/component/rot/cfpa
20982026-03-10T20:08:08.192ZINFOtest_retry_backoff (BackgroundTasks): found new target blueprint (first find) background_task = blueprint_loader target_id = 6e43be5e-eba9-4a79-84a8-7f9521f2c946 time_created = 2026-03-10 20:08:06.872722 UTC
20992026-03-10T20:08:08.192ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 301.381334ms iteration = 2 status = {"enabled":false,"status":"first target blueprint","target_id":"6e43be5e-eba9-4a79-84a8-7f9521f2c946","time_created":"2026-03-10T20:08:06.872722Z","time_found":"2026-03-10T20:08:08.192773535Z"}
21002026-03-10T20:08:08.192ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = blueprint_planner iteration = 3 reason = Dependency
21012026-03-10T20:08:08.192ZDEBGtest_retry_backoff (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
21022026-03-10T20:08:08.192ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 20.316\u{b5}s iteration = 3 status = "Disabled"
21032026-03-10T20:08:08.192ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 2 reason = Dependency
21042026-03-10T20:08:08.193ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = crdb_node_id_collector resource = Fleet
21052026-03-10T20:08:08.193ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = crdb_node_id_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21062026-03-10T20:08:08.193ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = crdb_node_id_collector resource = Database
21072026-03-10T20:08:08.193ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
21082026-03-10T20:08:08.193ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
21092026-03-10T20:08:08.195ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = blueprint_executor iteration = 2 reason = Dependency
21102026-03-10T20:08:08.195ZDEBGtest_retry_backoff (BackgroundTasks): blueprint execution: quiesce check background_task = blueprint_executor quiescing = false
21112026-03-10T20:08:08.195ZINFOtest_retry_backoff (NexusQuiesceHandle): initial state state = Running
21122026-03-10T20:08:08.195ZINFOtest_retry_backoff (SagaQuiesceHandle): initial quiesce state initial_state = Allowed
21132026-03-10T20:08:08.195ZWARNtest_retry_backoff (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = blueprint disabled target_id = 6e43be5e-eba9-4a79-84a8-7f9521f2c946
21142026-03-10T20:08:08.195ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 74.489\u{b5}s iteration = 2 status = {"enabled":false,"target_id":"6e43be5e-eba9-4a79-84a8-7f9521f2c946"}
21152026-03-10T20:08:08.195ZTRACtest_retry_backoff (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3 (built_in_user), .. }) resource = Database
21162026-03-10T20:08:08.195ZDEBGtest_retry_backoff (ServiceBalancer): roles roles = RoleSet { roles: {} }
21172026-03-10T20:08:08.196ZDEBGtest_retry_backoff (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3 (built_in_user), .. }) resource = Database result = Ok(())
21182026-03-10T20:08:08.196ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61c6d869-0096-4015-ae04-b896b091e4bc", "content-length": "727", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21192026-03-10T20:08:08.197ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0/component/rot/cfpa
21202026-03-10T20:08:08.197ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c48dddf7-505d-492d-acf0-173516467553", "content-length": "729", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21212026-03-10T20:08:08.197ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/0/component/rot/cfpa
21222026-03-10T20:08:08.198ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21232026-03-10T20:08:08.201ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
21242026-03-10T20:08:08.201ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = crdb_node_id_collector resource = Database
21252026-03-10T20:08:08.201ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
21262026-03-10T20:08:08.201ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
21272026-03-10T20:08:08.202ZTRACtest_retry_backoff (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3 (built_in_user), .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
21282026-03-10T20:08:08.202ZTRACtest_retry_backoff (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21292026-03-10T20:08:08.202ZTRACtest_retry_backoff (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3 (built_in_user), .. }) resource = Database
21302026-03-10T20:08:08.202ZDEBGtest_retry_backoff (ServiceBalancer): roles roles = RoleSet { roles: {} }
21312026-03-10T20:08:08.202ZDEBGtest_retry_backoff (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3 (built_in_user), .. }) resource = Database result = Ok(())
21322026-03-10T20:08:08.204ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa1b25d7-b637-41f5-a9b0-397f94506222", "content-length": "728", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21332026-03-10T20:08:08.204ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1
21342026-03-10T20:08:08.204ZDEBGtest_retry_backoff (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21352026-03-10T20:08:08.208ZDEBGtest_retry_backoff (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3 (built_in_user), .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
21362026-03-10T20:08:08.208ZINFOtest_retry_backoff (nexus): Rack initialized
21372026-03-10T20:08:08.208ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
21382026-03-10T20:08:08.208ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
21392026-03-10T20:08:08.209ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
21402026-03-10T20:08:08.246ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[::1]:32222/node/id
21412026-03-10T20:08:08.246ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = crdb_node_id_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:32222/node/id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:32222, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
21422026-03-10T20:08:08.247ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 54.249917ms iteration = 2 status = {"errors":[{"err":"failed to fetch node ID for zone f189113c-85eb-4d6a-b2fc-46ba3de160e6 at http://[::1]:32222: Communication Error: error sending request for url (http://[::1]:32222/node/id): error sending request for url (http://[::1]:32222/node/id): client error (Connect): tcp connect error: Connection refused (os error 146)","zone_id":"f189113c-85eb-4d6a-b2fc-46ba3de160e6"}],"nsuccess":0}
21432026-03-10T20:08:08.247ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f29f4c85-e5e3-48c9-a031-967cc06f44d3", "content-length": "734", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21442026-03-10T20:08:08.248ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1/component/sp/caboose?firmware_slot=0
21452026-03-10T20:08:08.248ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
21462026-03-10T20:08:08.248ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
21472026-03-10T20:08:08.248ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
21482026-03-10T20:08:08.250ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6118225-5212-4b80-9f0f-09a28498457e", "content-length": "111", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21492026-03-10T20:08:08.250ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1/component/sp/caboose?firmware_slot=1
21502026-03-10T20:08:08.250ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
21512026-03-10T20:08:08.250ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
21522026-03-10T20:08:08.250ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
21532026-03-10T20:08:08.252ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22db2bc3-108e-4fcb-9cc1-e4058c4c7af8", "content-length": "111", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21542026-03-10T20:08:08.252ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1/component/rot/caboose?firmware_slot=0
21552026-03-10T20:08:08.252ZTRACtest_retry_backoff (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21562026-03-10T20:08:08.252ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21572026-03-10T20:08:08.252ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
21582026-03-10T20:08:08.252ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
21592026-03-10T20:08:08.253ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
21602026-03-10T20:08:08.254ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2ac07cc-8377-4a24-8f63-911484737acc", "content-length": "163", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21612026-03-10T20:08:08.254ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1/component/rot/caboose?firmware_slot=1
21622026-03-10T20:08:08.254ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21632026-03-10T20:08:08.254ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
21642026-03-10T20:08:08.254ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
21652026-03-10T20:08:08.254ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
21662026-03-10T20:08:08.256ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47593c12-c88b-44a3-bb74-e87b1ad3c0a2", "content-length": "163", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21672026-03-10T20:08:08.256ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1/component/stage0/caboose?firmware_slot=0
21682026-03-10T20:08:08.256ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21692026-03-10T20:08:08.256ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
21702026-03-10T20:08:08.256ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
21712026-03-10T20:08:08.256ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
21722026-03-10T20:08:08.258ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1b0e3fb-80a5-4e90-b0cb-5a834322eaed", "content-length": "166", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21732026-03-10T20:08:08.258ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1/component/stage0/caboose?firmware_slot=1
21742026-03-10T20:08:08.258ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21752026-03-10T20:08:08.263ZDEBGtest_retry_backoff (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21762026-03-10T20:08:08.263ZTRACtest_retry_backoff (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21772026-03-10T20:08:08.263ZTRACtest_retry_backoff (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21782026-03-10T20:08:08.263ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21792026-03-10T20:08:08.263ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
21802026-03-10T20:08:08.263ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
21812026-03-10T20:08:08.263ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
21822026-03-10T20:08:08.267ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fe8fc4b-6e8f-4d38-87ba-5224ce19aadc", "content-length": "166", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21832026-03-10T20:08:08.267ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1/component/rot/cmpa
21842026-03-10T20:08:08.268ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21852026-03-10T20:08:08.268ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
21862026-03-10T20:08:08.268ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
21872026-03-10T20:08:08.268ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
21882026-03-10T20:08:08.269ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2ec07b0-05b7-4a7b-8d9d-b0f401e036de", "content-length": "702", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21892026-03-10T20:08:08.269ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1/component/rot/cfpa
21902026-03-10T20:08:08.270ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4089a82b-ba07-4fda-9dd6-74de44a2e2ca", "content-length": "727", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21912026-03-10T20:08:08.270ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1/component/rot/cfpa
21922026-03-10T20:08:08.270ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21932026-03-10T20:08:08.270ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
21942026-03-10T20:08:08.270ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
21952026-03-10T20:08:08.270ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
21962026-03-10T20:08:08.271ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc8fe737-e15d-472c-a7e1-795bd5c403aa", "content-length": "729", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21972026-03-10T20:08:08.271ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/switch/1/component/rot/cfpa
21982026-03-10T20:08:08.272ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "afb54550-7e48-492b-847c-361686faea1c", "content-length": "728", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
21992026-03-10T20:08:08.272ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0
22002026-03-10T20:08:08.272ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22012026-03-10T20:08:08.277ZDEBGtest_retry_backoff (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
22022026-03-10T20:08:08.277ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22032026-03-10T20:08:08.277ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22042026-03-10T20:08:08.278ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22052026-03-10T20:08:08.279ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "279178c8-c2b6-4fd6-a6d8-68d01f93d9b7", "content-length": "723", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22062026-03-10T20:08:08.279ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/host-boot-flash/active-slot
22072026-03-10T20:08:08.280ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/host-boot-flash/active-slot", status: 200, headers: {"content-type": "application/json", "x-request-id": "80b92e4b-3420-411d-8657-13dd8d57b70f", "content-length": "10", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22082026-03-10T20:08:08.280ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/host-boot-flash/hash/0
22092026-03-10T20:08:08.280ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22102026-03-10T20:08:08.280ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22112026-03-10T20:08:08.280ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22122026-03-10T20:08:08.281ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/host-boot-flash/hash/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c21946e-2166-4b80-b999-758369d76e84", "content-length": "144", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22132026-03-10T20:08:08.282ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/host-boot-flash/hash/1
22142026-03-10T20:08:08.282ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/host-boot-flash/hash/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0307332f-124d-418e-b42f-f50b0231350e", "content-length": "147", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22152026-03-10T20:08:08.282ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/sp/caboose?firmware_slot=0
22162026-03-10T20:08:08.282ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22172026-03-10T20:08:08.283ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22182026-03-10T20:08:08.283ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22192026-03-10T20:08:08.284ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "534fa7c3-f8c4-4b4c-9b70-139e4598dc8e", "content-length": "109", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22202026-03-10T20:08:08.284ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/sp/caboose?firmware_slot=1
22212026-03-10T20:08:08.284ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22222026-03-10T20:08:08.284ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22232026-03-10T20:08:08.285ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22242026-03-10T20:08:08.286ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "986dcf79-d502-42ac-b578-6f42824762f2", "content-length": "109", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22252026-03-10T20:08:08.286ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/rot/caboose?firmware_slot=0
22262026-03-10T20:08:08.286ZTRACtest_retry_backoff (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
22272026-03-10T20:08:08.286ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22282026-03-10T20:08:08.286ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22292026-03-10T20:08:08.286ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22302026-03-10T20:08:08.287ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22312026-03-10T20:08:08.288ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cece9b9-84ba-4277-9f3c-fee8582c084a", "content-length": "163", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22322026-03-10T20:08:08.288ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/rot/caboose?firmware_slot=1
22332026-03-10T20:08:08.288ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22342026-03-10T20:08:08.288ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22352026-03-10T20:08:08.288ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22362026-03-10T20:08:08.289ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22372026-03-10T20:08:08.290ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8339762-fae5-4065-9a9c-0010431dfed5", "content-length": "163", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22382026-03-10T20:08:08.290ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/stage0/caboose?firmware_slot=0
22392026-03-10T20:08:08.290ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22402026-03-10T20:08:08.290ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22412026-03-10T20:08:08.290ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22422026-03-10T20:08:08.291ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22432026-03-10T20:08:08.292ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fc5ba49-0f69-4ba1-a8ab-4ce613f1fb76", "content-length": "166", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22442026-03-10T20:08:08.292ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/stage0/caboose?firmware_slot=1
22452026-03-10T20:08:08.292ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22462026-03-10T20:08:08.298ZDEBGtest_retry_backoff (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
22472026-03-10T20:08:08.298ZTRACtest_retry_backoff (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
22482026-03-10T20:08:08.298ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22492026-03-10T20:08:08.298ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22502026-03-10T20:08:08.299ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22512026-03-10T20:08:08.302ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b98dc2a-9c66-4617-b441-dac68bae098d", "content-length": "166", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22522026-03-10T20:08:08.302ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/rot/cmpa
22532026-03-10T20:08:08.302ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22542026-03-10T20:08:08.302ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22552026-03-10T20:08:08.303ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22562026-03-10T20:08:08.304ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0bd08f5-5813-46ff-b27d-361628d06d14", "content-length": "702", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22572026-03-10T20:08:08.304ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/rot/cfpa
22582026-03-10T20:08:08.304ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "738c37c9-69f4-4331-b90b-d4918ed628a6", "content-length": "727", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22592026-03-10T20:08:08.305ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/rot/cfpa
22602026-03-10T20:08:08.305ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22612026-03-10T20:08:08.305ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22622026-03-10T20:08:08.305ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22632026-03-10T20:08:08.306ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e408e686-025e-41cc-9d16-ee85000abb23", "content-length": "729", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22642026-03-10T20:08:08.306ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/0/component/rot/cfpa
22652026-03-10T20:08:08.307ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78f85914-7537-4c46-a1fe-e8cd63504809", "content-length": "728", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22662026-03-10T20:08:08.307ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1
22672026-03-10T20:08:08.307ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22682026-03-10T20:08:08.307ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22692026-03-10T20:08:08.308ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22702026-03-10T20:08:08.309ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47b4de37-e3c6-40a4-aec7-9eede0784eb8", "content-length": "723", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22712026-03-10T20:08:08.309ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/host-boot-flash/active-slot
22722026-03-10T20:08:08.309ZTRACtest_retry_backoff (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22732026-03-10T20:08:08.309ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22742026-03-10T20:08:08.309ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22752026-03-10T20:08:08.309ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22762026-03-10T20:08:08.310ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22772026-03-10T20:08:08.310ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/host-boot-flash/active-slot", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a92339e-4b2d-4c51-ad87-d5ddf4ec8418", "content-length": "10", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22782026-03-10T20:08:08.310ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/host-boot-flash/hash/0
22792026-03-10T20:08:08.311ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/host-boot-flash/hash/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b744cbd-091e-4991-a2e8-aaf45cb7cea4", "content-length": "144", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22802026-03-10T20:08:08.311ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/host-boot-flash/hash/1
22812026-03-10T20:08:08.311ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22822026-03-10T20:08:08.311ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22832026-03-10T20:08:08.311ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22842026-03-10T20:08:08.311ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22852026-03-10T20:08:08.312ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/host-boot-flash/hash/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "129abef4-18bd-4366-b4dd-779b2df8fde0", "content-length": "147", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22862026-03-10T20:08:08.312ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/sp/caboose?firmware_slot=0
22872026-03-10T20:08:08.313ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22882026-03-10T20:08:08.313ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22892026-03-10T20:08:08.313ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
22902026-03-10T20:08:08.313ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
22912026-03-10T20:08:08.314ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7ad16f4-659d-44fc-b8f9-713ecf356ba0", "content-length": "109", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
22922026-03-10T20:08:08.314ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/sp/caboose?firmware_slot=1
22932026-03-10T20:08:08.315ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22942026-03-10T20:08:08.321ZDEBGtest_retry_backoff (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22952026-03-10T20:08:08.321ZTRACtest_retry_backoff (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22962026-03-10T20:08:08.321ZTRACtest_retry_backoff (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22972026-03-10T20:08:08.321ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22982026-03-10T20:08:08.321ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
22992026-03-10T20:08:08.321ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23002026-03-10T20:08:08.321ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23012026-03-10T20:08:08.322ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17860373-f99a-4769-9e33-547e67c046f4", "content-length": "109", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23022026-03-10T20:08:08.322ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/rot/caboose?firmware_slot=0
23032026-03-10T20:08:08.322ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
23042026-03-10T20:08:08.322ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
23052026-03-10T20:08:08.322ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23062026-03-10T20:08:08.323ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23072026-03-10T20:08:08.324ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e77e649-a498-4e3d-8918-1a1ab8fad107", "content-length": "163", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23082026-03-10T20:08:08.324ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/rot/caboose?firmware_slot=1
23092026-03-10T20:08:08.324ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23102026-03-10T20:08:08.324ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
23112026-03-10T20:08:08.324ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23122026-03-10T20:08:08.325ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23132026-03-10T20:08:08.326ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e02f4008-e97d-421b-8d1f-b683d3d709ed", "content-length": "163", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23142026-03-10T20:08:08.326ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/stage0/caboose?firmware_slot=0
23152026-03-10T20:08:08.326ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23162026-03-10T20:08:08.332ZDEBGtest_retry_backoff (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
23172026-03-10T20:08:08.332ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
23182026-03-10T20:08:08.332ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23192026-03-10T20:08:08.332ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23202026-03-10T20:08:08.334ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1638aa2b-e6c6-44ba-abb1-f3f989cbd8a9", "content-length": "166", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23212026-03-10T20:08:08.334ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/stage0/caboose?firmware_slot=1
23222026-03-10T20:08:08.335ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4a475b4-5761-4624-be32-37c4611dd603", "content-length": "166", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23232026-03-10T20:08:08.335ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/rot/cmpa
23242026-03-10T20:08:08.335ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
23252026-03-10T20:08:08.335ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23262026-03-10T20:08:08.336ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23272026-03-10T20:08:08.336ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "345ddf96-0033-42fa-aae4-d0d6a6640726", "content-length": "702", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23282026-03-10T20:08:08.336ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/rot/cfpa
23292026-03-10T20:08:08.337ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "704d0e2d-32b4-4b63-930a-880de854cea8", "content-length": "727", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23302026-03-10T20:08:08.337ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/rot/cfpa
23312026-03-10T20:08:08.337ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
23322026-03-10T20:08:08.337ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23332026-03-10T20:08:08.338ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23342026-03-10T20:08:08.339ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79527fe6-4a80-4c4f-8202-5f4226395bd8", "content-length": "729", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23352026-03-10T20:08:08.339ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/sp/sled/1/component/rot/cfpa
23362026-03-10T20:08:08.340ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5303fd62-4fe8-45f6-ba16-8005eeccedaa", "content-length": "728", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23372026-03-10T20:08:08.340ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Fleet
23382026-03-10T20:08:08.340ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23392026-03-10T20:08:08.340ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
23402026-03-10T20:08:08.340ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23412026-03-10T20:08:08.340ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
23422026-03-10T20:08:08.341ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
23432026-03-10T20:08:08.341ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23442026-03-10T20:08:08.341ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23452026-03-10T20:08:08.342ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23462026-03-10T20:08:08.345ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Fleet result = Ok(())
23472026-03-10T20:08:08.346ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Fleet
23482026-03-10T20:08:08.346ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23492026-03-10T20:08:08.346ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
23502026-03-10T20:08:08.346ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23512026-03-10T20:08:08.346ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
23522026-03-10T20:08:08.346ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
23532026-03-10T20:08:08.347ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23542026-03-10T20:08:08.347ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23552026-03-10T20:08:08.348ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23562026-03-10T20:08:08.351ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Fleet result = Ok(())
23572026-03-10T20:08:08.351ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
23582026-03-10T20:08:08.351ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23592026-03-10T20:08:08.351ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
23602026-03-10T20:08:08.352ZTRACtest_retry_backoff (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
23612026-03-10T20:08:08.352ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
23622026-03-10T20:08:08.352ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
23632026-03-10T20:08:08.352ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23642026-03-10T20:08:08.352ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23652026-03-10T20:08:08.354ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
23662026-03-10T20:08:08.354ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
23672026-03-10T20:08:08.354ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23682026-03-10T20:08:08.354ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23692026-03-10T20:08:08.392ZDEBGtest_retry_backoff (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:45969
23702026-03-10T20:08:08.392ZDEBGtest_retry_backoff (BackgroundTasks): client request SledAgent = http://[::1]:45969 background_task = inventory_collection body = None method = GET uri = http://[::1]:45969/inventory
23712026-03-10T20:08:08.393ZINFOtest_retry_backoff (dropshot): accepted connection local_addr = [::1]:45969 remote_addr = [::1]:57254 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23722026-03-10T20:08:08.396ZTRACtest_retry_backoff (dropshot): incoming request local_addr = [::1]:45969 method = GET remote_addr = [::1]:57254 req_id = a69da5c2-af56-41d5-9432-d150af7f45a2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
23732026-03-10T20:08:08.396ZDEBGtest_retry_backoff (dropshot): determined request API version local_addr = [::1]:45969 method = GET remote_addr = [::1]:57254 req_id = a69da5c2-af56-41d5-9432-d150af7f45a2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory version = 27.0.0
23742026-03-10T20:08:08.398ZINFOtest_retry_backoff (dropshot): request completed latency_us = 5247 local_addr = [::1]:45969 method = GET remote_addr = [::1]:57254 req_id = a69da5c2-af56-41d5-9432-d150af7f45a2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
23752026-03-10T20:08:08.398ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23762026-03-10T20:08:08.398ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
23772026-03-10T20:08:08.398ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
23782026-03-10T20:08:08.399ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
23792026-03-10T20:08:08.400ZDEBGtest_retry_backoff (BackgroundTasks): client response SledAgent = http://[::1]:45969 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:45969/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a69da5c2-af56-41d5-9432-d150af7f45a2", "content-length": "20607", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23802026-03-10T20:08:08.402ZDEBGtest_retry_backoff (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
23812026-03-10T20:08:08.402ZDEBGtest_retry_backoff (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
23822026-03-10T20:08:08.402ZDEBGtest_retry_backoff (BackgroundTasks): begin collection from CockroachDB nodes background_task = inventory_collection
23832026-03-10T20:08:08.402ZDEBGtest_retry_backoff (CockroachClusterAdminClient): client request body = None method = GET uri = http://[::1]:32222/node/id
23842026-03-10T20:08:08.402ZDEBGtest_retry_backoff (CockroachClusterAdminClient): client response result = Err(reqwest::Error { kind: Request, url: "http://[::1]:32222/node/id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:32222, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
23852026-03-10T20:08:08.402ZWARNtest_retry_backoff (CockroachClusterAdminClient): Failed to fetch metrics from CockroachDB node error = Failed to get node ID from [::1]:32222
23862026-03-10T20:08:08.438ZDEBGtest_retry_backoff (BackgroundTasks): begin collection from NTP admin (timesync) background_task = inventory_collection sled_agent_url = http://[::1]:10123 zone_id = 5c201080-cc41-4511-8008-eb7a16cba8eb (omicron_zone)
23872026-03-10T20:08:08.439ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None method = GET ntp_admin_url = http://[::1]:10123 uri = http://[::1]:10123/timesync
23882026-03-10T20:08:08.439ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection ntp_admin_url = http://[::1]:10123 result = Err(reqwest::Error { kind: Request, url: "http://[::1]:10123/timesync", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:10123, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
23892026-03-10T20:08:08.439ZDEBGtest_retry_backoff (BackgroundTasks): begin collection from internal DNS servers background_task = inventory_collection
23902026-03-10T20:08:08.475ZDEBGtest_retry_backoff (BackgroundTasks): begin collection from DNS server background_task = inventory_collection zone_id = 81796aa6-4ecc-4d56-a811-dc112da8da6c (omicron_zone)
23912026-03-10T20:08:08.475ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = inventory_collection body = None internal_dns_url = http://[::1]:64860 method = GET uri = http://[::1]:64860/config
23922026-03-10T20:08:08.475ZINFOtest_retry_backoff (http): accepted connection kind = dns local_addr = [::1]:64860 remote_addr = [::1]:63749
23932026-03-10T20:08:08.475ZTRACtest_retry_backoff (http): incoming request kind = dns local_addr = [::1]:64860 method = GET remote_addr = [::1]:63749 req_id = a7c0c834-0aef-41b0-a695-729b5e44d3d2 uri = /config
23942026-03-10T20:08:08.475ZDEBGtest_retry_backoff (http): determined request API version kind = dns local_addr = [::1]:64860 method = GET remote_addr = [::1]:63749 req_id = a7c0c834-0aef-41b0-a695-729b5e44d3d2 uri = /config version = 2.0.0
23952026-03-10T20:08:08.476ZINFOtest_retry_backoff (http): request completed kind = dns latency_us = 573 local_addr = [::1]:64860 method = GET remote_addr = [::1]:63749 req_id = a7c0c834-0aef-41b0-a695-729b5e44d3d2 response_code = 200 uri = /config
23962026-03-10T20:08:08.476ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = inventory_collection internal_dns_url = http://[::1]:64860 result = Ok(Response { url: "http://[::1]:64860/config", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7c0c834-0aef-41b0-a695-729b5e44d3d2", "content-length": "3478", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
23972026-03-10T20:08:08.477ZDEBGtest_retry_backoff (BackgroundTasks): finished collection from DNS server background_task = inventory_collection zone_id = 81796aa6-4ecc-4d56-a811-dc112da8da6c (omicron_zone)
23982026-03-10T20:08:08.477ZDEBGtest_retry_backoff (BackgroundTasks): finished collection from internal DNS servers background_task = inventory_collection
23992026-03-10T20:08:08.477ZDEBGtest_retry_backoff (BackgroundTasks): finished collection background_task = inventory_collection
24002026-03-10T20:08:08.477ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Inventory
24012026-03-10T20:08:08.477ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
24022026-03-10T20:08:08.477ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
24032026-03-10T20:08:08.477ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
24042026-03-10T20:08:08.477ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
24052026-03-10T20:08:08.478ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24062026-03-10T20:08:08.484ZDEBGtest_retry_backoff (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
24072026-03-10T20:08:08.484ZTRACtest_retry_backoff (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
24082026-03-10T20:08:08.484ZTRACtest_retry_backoff (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
24092026-03-10T20:08:08.484ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
24102026-03-10T20:08:08.484ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
24112026-03-10T20:08:08.484ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
24122026-03-10T20:08:08.484ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
24132026-03-10T20:08:08.486ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24142026-03-10T20:08:08.489ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Inventory result = Ok(())
24152026-03-10T20:08:08.490ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database
24162026-03-10T20:08:08.490ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
24172026-03-10T20:08:08.490ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_collection resource = Database result = Ok(())
24182026-03-10T20:08:08.491ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
24192026-03-10T20:08:08.491ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
24202026-03-10T20:08:08.491ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
24212026-03-10T20:08:08.491ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
24222026-03-10T20:08:08.493ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
24232026-03-10T20:08:08.493ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
24242026-03-10T20:08:08.493ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
24252026-03-10T20:08:08.493ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
24262026-03-10T20:08:08.495ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24272026-03-10T20:08:08.501ZDEBGtest_retry_backoff (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
24282026-03-10T20:08:08.501ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
24292026-03-10T20:08:08.501ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
24302026-03-10T20:08:08.501ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
24312026-03-10T20:08:08.505ZDEBGtest_retry_backoff (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 966b4c74-e251-41b7-ba6b-c2516cebdadf, kind: Service { id: 61d3324b-3a36-494c-95ab-02b65214832c }, name: Name("external-dns-61d3324b-3a36-494c-95ab-02b65214832c"), ip_config: V6(PrivateIpv6Config { ip: fd77:e9d2:9cd9:1::6, subnet: Ipv6Net { addr: fd77:e9d2:9cd9:1::, width: 64 }, transit_ips: [] }), mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), vni: Vni(100), primary: true, slot: 0 }], Name("nexus"): [NetworkInterface { id: 46219700-2d95-43d8-ae72-bed7bc0ef377, kind: Service { id: 913233fe-92a8-4635-9572-183f495429c4 }, name: Name("nexus-913233fe-92a8-4635-9572-183f495429c4"), ip_config: V4(PrivateIpv4Config { ip: 172.30.2.6, subnet: Ipv4Net { addr: 172.30.2.0, width: 24 }, transit_ips: [] }), mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), vni: Vni(100), primary: true, slot: 0 }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
24322026-03-10T20:08:08.505ZTRACtest_retry_backoff (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet
24332026-03-10T20:08:08.505ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
24342026-03-10T20:08:08.505ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
24352026-03-10T20:08:08.505ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
24362026-03-10T20:08:08.506ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
24372026-03-10T20:08:08.507ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24382026-03-10T20:08:08.510ZDEBGtest_retry_backoff (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Fleet result = Ok(())
24392026-03-10T20:08:08.510ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
24402026-03-10T20:08:08.510ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
24412026-03-10T20:08:08.510ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
24422026-03-10T20:08:08.511ZTRACtest_retry_backoff (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2026-03-10T19:22:50.332390Z, time_modified: 2026-03-10T20:08:07.852989Z, allowed_ips: None }
24432026-03-10T20:08:08.512ZDEBGtest_retry_backoff (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
24442026-03-10T20:08:08.512ZDEBGtest_retry_backoff (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 966b4c74-e251-41b7-ba6b-c2516cebdadf, kind: Service { id: 61d3324b-3a36-494c-95ab-02b65214832c }, name: Name("external-dns-61d3324b-3a36-494c-95ab-02b65214832c"), ip_config: V6(PrivateIpv6Config { ip: fd77:e9d2:9cd9:1::6, subnet: Ipv6Net { addr: fd77:e9d2:9cd9:1::, width: 64 }, transit_ips: [] }), mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46219700-2d95-43d8-ae72-bed7bc0ef377, kind: Service { id: 913233fe-92a8-4635-9572-183f495429c4 }, name: Name("nexus-913233fe-92a8-4635-9572-183f495429c4"), ip_config: V4(PrivateIpv4Config { ip: 172.30.2.6, subnet: Ipv4Net { addr: 172.30.2.0, width: 24 }, transit_ips: [] }), mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 966b4c74-e251-41b7-ba6b-c2516cebdadf, kind: Service { id: 61d3324b-3a36-494c-95ab-02b65214832c }, name: Name("external-dns-61d3324b-3a36-494c-95ab-02b65214832c"), ip_config: V6(PrivateIpv6Config { ip: fd77:e9d2:9cd9:1::6, subnet: Ipv6Net { addr: fd77:e9d2:9cd9:1::, width: 64 }, transit_ips: [] }), mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp(Some(VpcFirewallIcmpFilter { icmp_type: 3, code: Some(IcmpParamRange { first: 3, last: 4 }) })), Icmp(Some(VpcFirewallIcmpFilter { icmp_type: 5, code: None })), Icmp(Some(VpcFirewallIcmpFilter { icmp_type: 11, code: None }))]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46219700-2d95-43d8-ae72-bed7bc0ef377, kind: Service { id: 913233fe-92a8-4635-9572-183f495429c4 }, name: Name("nexus-913233fe-92a8-4635-9572-183f495429c4"), ip_config: V4(PrivateIpv4Config { ip: 172.30.2.6, subnet: Ipv4Net { addr: 172.30.2.0, width: 24 }, transit_ips: [] }), mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }]
24452026-03-10T20:08:08.512ZDEBGtest_retry_backoff (InternalApi): resolved 3 rules for sleds
24462026-03-10T20:08:08.523ZDEBGtest_retry_backoff (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229 (sled), time_created: 2026-03-10T20:08:06.085399Z, time_modified: 2026-03-10T20:08:06.085399Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "SimGimlet00", part_number: "i86pc", revision: SqlU32(3), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(45969), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(47851), cpu_family: AmdMilan }]
24472026-03-10T20:08:08.523ZDEBGtest_retry_backoff (InternalApi): sending firewall rules to sled agents
24482026-03-10T20:08:08.565ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
24492026-03-10T20:08:08.565ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
24502026-03-10T20:08:08.566ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
24512026-03-10T20:08:08.570ZTRACtest_retry_backoff (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229 (sled), lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
24522026-03-10T20:08:08.570ZTRACtest_retry_backoff (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
24532026-03-10T20:08:08.570ZTRACtest_retry_backoff (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database
24542026-03-10T20:08:08.570ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {} }
24552026-03-10T20:08:08.571ZDEBGtest_retry_backoff (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Database result = Ok(())
24562026-03-10T20:08:08.572ZDEBGtest_retry_backoff (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24572026-03-10T20:08:08.577ZDEBGtest_retry_backoff (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229 (sled), lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
24582026-03-10T20:08:08.577ZDEBGtest_retry_backoff (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:45969/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24592026-03-10T20:08:08.577ZINFOtest_retry_backoff (dropshot): accepted connection local_addr = [::1]:45969 remote_addr = [::1]:39157 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
24602026-03-10T20:08:08.578ZTRACtest_retry_backoff (dropshot): incoming request local_addr = [::1]:45969 method = PUT remote_addr = [::1]:39157 req_id = f4e82d04-6b46-417a-9b9a-e21e4cd98f16 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24612026-03-10T20:08:08.578ZDEBGtest_retry_backoff (dropshot): determined request API version local_addr = [::1]:45969 method = PUT remote_addr = [::1]:39157 req_id = f4e82d04-6b46-417a-9b9a-e21e4cd98f16 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules version = 27.0.0
24622026-03-10T20:08:08.579ZINFOtest_retry_backoff (dropshot): request completed latency_us = 1141 local_addr = [::1]:45969 method = PUT remote_addr = [::1]:39157 req_id = f4e82d04-6b46-417a-9b9a-e21e4cd98f16 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24632026-03-10T20:08:08.579ZDEBGtest_retry_backoff (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:45969/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "f4e82d04-6b46-417a-9b9a-e21e4cd98f16", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
24642026-03-10T20:08:08.579ZINFOtest_retry_backoff (InternalApi): updated firewall rules on 1 sleds
24652026-03-10T20:08:08.579ZINFOtest_retry_backoff (nexus): plumbed initial IP allowlist
24662026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = / versions = all
24672026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /access versions = all
24682026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /assets/{path} versions = all
24692026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /device/auth versions = all
24702026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /device/confirm versions = all
24712026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /device/success versions = all
24722026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /device/token versions = all
24732026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /device/verify versions = all
24742026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /experimental/v1/probes versions = all starting from 2026010300.0.0
24752026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /experimental/v1/probes versions = all ending with 2026010300.0.0
24762026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /experimental/v1/probes versions = all starting from 2026010500.0.0
24772026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /experimental/v1/probes versions = all ending with 2026010500.0.0
24782026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /experimental/v1/probes/{probe} versions = all
24792026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /experimental/v1/probes/{probe} versions = all starting from 2026010300.0.0
24802026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /experimental/v1/probes/{probe} versions = all ending with 2026010300.0.0
24812026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /experimental/v1/system/support-bundles versions = all
24822026-03-10T20:08:08.618ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /experimental/v1/system/support-bundles versions = all
24832026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /experimental/v1/system/support-bundles/{bundle_id} versions = all
24842026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /experimental/v1/system/support-bundles/{bundle_id} versions = all
24852026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /experimental/v1/system/support-bundles/{bundle_id} versions = all
24862026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /experimental/v1/system/support-bundles/{bundle_id}/download versions = all
24872026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = HEAD path = /experimental/v1/system/support-bundles/{bundle_id}/download versions = all
24882026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /experimental/v1/system/support-bundles/{bundle_id}/download/{file} versions = all
24892026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = HEAD path = /experimental/v1/system/support-bundles/{bundle_id}/download/{file} versions = all
24902026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /experimental/v1/system/support-bundles/{bundle_id}/index versions = all
24912026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /images versions = all
24922026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /login versions = all
24932026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /login/{silo_name}/local versions = all
24942026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
24952026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
24962026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
24972026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /lookup/{path} versions = all
24982026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /projects/{path} versions = all
24992026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /projects-new versions = all
25002026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /scim/v2/Groups versions = all
25012026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /scim/v2/Groups versions = all
25022026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /scim/v2/Groups/{group_id} versions = all
25032026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /scim/v2/Groups/{group_id} versions = all
25042026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PATCH path = /scim/v2/Groups/{group_id} versions = all
25052026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /scim/v2/Groups/{group_id} versions = all
25062026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /scim/v2/Users versions = all
25072026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /scim/v2/Users versions = all
25082026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /scim/v2/Users/{user_id} versions = all
25092026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /scim/v2/Users/{user_id} versions = all
25102026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PATCH path = /scim/v2/Users/{user_id} versions = all
25112026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /scim/v2/Users/{user_id} versions = all
25122026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /settings/{path} versions = all
25132026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /system/{path} versions = all
25142026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /utilization versions = all
25152026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/affinity-groups versions = all
25162026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/affinity-groups versions = all
25172026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
25182026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
25192026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
25202026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
25212026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
25222026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
25232026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
25242026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/alert-classes versions = all
25252026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/alert-receivers versions = all
25262026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/alert-receivers/{receiver} versions = all
25272026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/alert-receivers/{receiver} versions = all
25282026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/alert-receivers/{receiver}/deliveries versions = all
25292026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/alert-receivers/{receiver}/probe versions = all
25302026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/alert-receivers/{receiver}/subscriptions versions = all
25312026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/alert-receivers/{receiver}/subscriptions/{subscription} versions = all
25322026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/alerts/{alert_id}/resend versions = all
25332026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/anti-affinity-groups versions = all
25342026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/anti-affinity-groups versions = all
25352026-03-10T20:08:08.619ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
25362026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
25372026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
25382026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
25392026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
25402026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
25412026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
25422026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/auth-settings versions = all
25432026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/auth-settings versions = all
25442026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/certificates versions = all
25452026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/certificates versions = all
25462026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/certificates/{certificate} versions = all
25472026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/certificates/{certificate} versions = all
25482026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/disks versions = all starting from 2026013001.0.0
25492026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/disks versions = from 2025120300.0.0 to 2026013001.0.0
25502026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/disks versions = all ending with 2025120300.0.0
25512026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/disks versions = all starting from 2026013100.0.0
25522026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/disks versions = from 2026013001.0.0 to 2026013100.0.0
25532026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/disks versions = from 2025120300.0.0 to 2026013001.0.0
25542026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/disks versions = all ending with 2025120300.0.0
25552026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/disks/{disk} versions = all
25562026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/disks/{disk} versions = all starting from 2026013001.0.0
25572026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/disks/{disk} versions = from 2025120300.0.0 to 2026013001.0.0
25582026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/disks/{disk} versions = all ending with 2025120300.0.0
25592026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/disks/{disk}/bulk-write versions = all
25602026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
25612026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
25622026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/disks/{disk}/finalize versions = all
25632026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/external-subnets versions = all starting from 2026011601.0.0
25642026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/external-subnets versions = all starting from 2026012201.0.0
25652026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/external-subnets versions = from 2026011601.0.0 to 2026012201.0.0
25662026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/external-subnets/{external_subnet} versions = all starting from 2026011601.0.0
25672026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/external-subnets/{external_subnet} versions = all starting from 2026011601.0.0
25682026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/external-subnets/{external_subnet} versions = all starting from 2026011601.0.0
25692026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/external-subnets/{external_subnet}/attach versions = all starting from 2026011601.0.0
25702026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/external-subnets/{external_subnet}/detach versions = all starting from 2026011601.0.0
25712026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/floating-ips versions = all
25722026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/floating-ips versions = all starting from 2026012200.0.0
25732026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/floating-ips versions = from 2026011600.0.0 to 2026012200.0.0
25742026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/floating-ips versions = from 2026010500.0.0 to 2026011600.0.0
25752026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/floating-ips versions = from 2025122300.0.0 to 2026010500.0.0
25762026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/floating-ips versions = all ending with 2025122300.0.0
25772026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
25782026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/floating-ips/{floating_ip} versions = all
25792026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
25802026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
25812026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
25822026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/groups versions = all starting from 2026030200.0.0
25832026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/groups versions = all ending with 2026030200.0.0
25842026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/groups/{group_id} versions = all starting from 2026030200.0.0
25852026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/groups/{group_id} versions = all ending with 2026030200.0.0
25862026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/images versions = all
25872026-03-10T20:08:08.620ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/images versions = all
25882026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/images/{image} versions = all
25892026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/images/{image} versions = all
25902026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/images/{image}/demote versions = all
25912026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/images/{image}/promote versions = all
25922026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances versions = all
25932026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances versions = all starting from 2026013100.0.0
25942026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances versions = from 2026013001.0.0 to 2026013100.0.0
25952026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances versions = from 2026010800.0.0 to 2026013001.0.0
25962026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances versions = from 2026010500.0.0 to 2026010800.0.0
25972026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances versions = from 2026010300.0.0 to 2026010500.0.0
25982026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances versions = from 2025122300.0.0 to 2026010300.0.0
25992026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances versions = from 2025120300.0.0 to 2025122300.0.0
26002026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances versions = all ending with 2025120300.0.0
26012026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/instances/{instance} versions = all
26022026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance} versions = all
26032026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/instances/{instance} versions = all starting from 2026010800.0.0
26042026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/instances/{instance} versions = all ending with 2026010800.0.0
26052026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
26062026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
26072026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/disks versions = all starting from 2026013001.0.0
26082026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/disks versions = from 2025120300.0.0 to 2026013001.0.0
26092026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/disks versions = all ending with 2025120300.0.0
26102026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/disks/attach versions = all starting from 2026013001.0.0
26112026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/disks/attach versions = from 2025120300.0.0 to 2026013001.0.0
26122026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/disks/attach versions = all ending with 2025120300.0.0
26132026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/disks/detach versions = all starting from 2026013001.0.0
26142026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/disks/detach versions = from 2025120300.0.0 to 2026013001.0.0
26152026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/disks/detach versions = all ending with 2025120300.0.0
26162026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/external-ips versions = all
26172026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all starting from 2026012300.0.0
26182026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all ending with 2026012300.0.0
26192026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all starting from 2026010500.0.0
26202026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = from 2025122300.0.0 to 2026010500.0.0
26212026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all ending with 2025122300.0.0
26222026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/external-subnets versions = all starting from 2026013000.0.0
26232026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/multicast-groups versions = all starting from 2026010800.0.0
26242026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/multicast-groups versions = all ending with 2026010800.0.0
26252026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/instances/{instance}/multicast-groups/{multicast_group} versions = all starting from 2026010800.0.0
26262026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/instances/{instance}/multicast-groups/{multicast_group} versions = all ending with 2026010800.0.0
26272026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/instances/{instance}/multicast-groups/{multicast_group} versions = all starting from 2026010800.0.0
26282026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/instances/{instance}/multicast-groups/{multicast_group} versions = all ending with 2026010800.0.0
26292026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/reboot versions = all
26302026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/serial-console versions = all
26312026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
26322026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
26332026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/start versions = all
26342026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/instances/{instance}/stop versions = all
26352026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/internet-gateway-ip-addresses versions = all
26362026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/internet-gateway-ip-addresses versions = all
26372026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
26382026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/internet-gateway-ip-pools versions = all
26392026-03-10T20:08:08.621ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/internet-gateway-ip-pools versions = all
26402026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
26412026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/internet-gateways versions = all
26422026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/internet-gateways versions = all
26432026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
26442026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/internet-gateways/{gateway} versions = all
26452026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/ip-pools versions = all starting from 2026020900.0.0
26462026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/ip-pools versions = from 2026010100.0.0 to 2026020900.0.0
26472026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/ip-pools versions = all ending with 2026010100.0.0
26482026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/ip-pools/{pool} versions = all starting from 2026020900.0.0
26492026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/ip-pools/{pool} versions = from 2026010100.0.0 to 2026020900.0.0
26502026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/ip-pools/{pool} versions = all ending with 2026010100.0.0
26512026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/login/{silo_name}/local versions = all
26522026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/logout versions = all
26532026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/me versions = all starting from 2026030200.0.0
26542026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/me versions = all ending with 2026030200.0.0
26552026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/me/access-tokens versions = all
26562026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/me/access-tokens/{token_id} versions = all
26572026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/me/groups versions = all starting from 2026030200.0.0
26582026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/me/groups versions = all ending with 2026030200.0.0
26592026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/me/ssh-keys versions = all
26602026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/me/ssh-keys versions = all
26612026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
26622026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
26632026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/metrics/{metric_name} versions = all
26642026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/multicast-groups versions = all starting from 2026010800.0.0
26652026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/multicast-groups versions = all ending with 2026010800.0.0
26662026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/multicast-groups versions = all ending with 2026010800.0.0
26672026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/multicast-groups/{multicast_group} versions = all ending with 2026010800.0.0
26682026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/multicast-groups/{multicast_group} versions = all starting from 2026010800.0.0
26692026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/multicast-groups/{multicast_group} versions = all ending with 2026010800.0.0
26702026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/multicast-groups/{multicast_group} versions = all ending with 2026010800.0.0
26712026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/multicast-groups/{multicast_group}/members versions = all starting from 2026010800.0.0
26722026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/multicast-groups/{multicast_group}/members versions = all ending with 2026010800.0.0
26732026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/multicast-groups/{multicast_group}/members versions = all ending with 2026010800.0.0
26742026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/multicast-groups/{multicast_group}/members/{instance} versions = all ending with 2026010800.0.0
26752026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/network-interfaces versions = all starting from 2026010300.0.0
26762026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/network-interfaces versions = all ending with 2026010300.0.0
26772026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/network-interfaces versions = all starting from 2026010300.0.0
26782026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/network-interfaces versions = all ending with 2026010300.0.0
26792026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/network-interfaces/{interface} versions = all
26802026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/network-interfaces/{interface} versions = all starting from 2026010300.0.0
26812026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/network-interfaces/{interface} versions = all ending with 2026010300.0.0
26822026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/network-interfaces/{interface} versions = all starting from 2026010300.0.0
26832026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/network-interfaces/{interface} versions = all ending with 2026010300.0.0
26842026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/ping versions = all
26852026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/policy versions = all
26862026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/policy versions = all
26872026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/projects versions = all
26882026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/projects versions = all
26892026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/projects/{project} versions = all
26902026-03-10T20:08:08.622ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/projects/{project} versions = all
26912026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/projects/{project} versions = all
26922026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/projects/{project}/policy versions = all
26932026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/projects/{project}/policy versions = all
26942026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/snapshots versions = all
26952026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/snapshots versions = all
26962026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/snapshots/{snapshot} versions = all
26972026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/snapshots/{snapshot} versions = all
26982026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/subnet-pools versions = all starting from 2026020900.0.0
26992026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/subnet-pools versions = from 2026020600.0.0 to 2026020900.0.0
27002026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/subnet-pools/{pool} versions = all starting from 2026020900.0.0
27012026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/audit-log versions = all
27022026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/disks versions = all
27032026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
27042026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
27052026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/racks versions = all
27062026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
27072026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/racks/{rack_id}/membership versions = all starting from 2026012100.0.0
27082026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/hardware/racks/{rack_id}/membership/abort versions = all starting from 2026020200.0.0
27092026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/hardware/racks/{rack_id}/membership/add versions = all starting from 2026012100.0.0
27102026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/sleds versions = all
27112026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/hardware/sleds versions = all ending with 2026021900.0.0
27122026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
27132026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
27142026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
27152026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
27162026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
27172026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/switch-port versions = all
27182026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27192026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27202026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
27212026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
27222026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
27232026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/switches versions = all
27242026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
27252026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/identity-providers versions = all
27262026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/identity-providers/local/users versions = all starting from 2026030200.0.0
27272026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/identity-providers/local/users versions = all ending with 2026030200.0.0
27282026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
27292026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
27302026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/identity-providers/saml versions = all
27312026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
27322026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools versions = all starting from 2026020900.0.0
27332026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools versions = all ending with 2026020900.0.0
27342026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools versions = all starting from 2026020900.0.0
27352026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools versions = all ending with 2026020900.0.0
27362026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/ip-pools/{pool} versions = all starting from 2026020900.0.0
27372026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/ip-pools/{pool} versions = all ending with 2026020900.0.0
27382026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools/{pool} versions = all starting from 2026020900.0.0
27392026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools/{pool} versions = all ending with 2026020900.0.0
27402026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/ip-pools/{pool} versions = all starting from 2026020900.0.0
27412026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/ip-pools/{pool} versions = all ending with 2026020900.0.0
27422026-03-10T20:08:08.623ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all starting from 2026020900.0.0
27432026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all ending with 2026020900.0.0
27442026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all starting from 2026020900.0.0
27452026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all ending with 2026020900.0.0
27462026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all starting from 2026020900.0.0
27472026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all ending with 2026020900.0.0
27482026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all starting from 2026020900.0.0
27492026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all ending with 2026020900.0.0
27502026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all starting from 2026020900.0.0
27512026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all ending with 2026020900.0.0
27522026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all starting from 2026020900.0.0
27532026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all ending with 2026020900.0.0
27542026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all starting from 2026020900.0.0
27552026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all ending with 2026020900.0.0
27562026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all starting from 2026020900.0.0
27572026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all ending with 2026020900.0.0
27582026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools-service versions = all starting from 2026020900.0.0
27592026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools-service versions = all ending with 2026020900.0.0
27602026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools-service/ranges versions = all starting from 2026020900.0.0
27612026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/ip-pools-service/ranges versions = all ending with 2026020900.0.0
27622026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all starting from 2026020900.0.0
27632026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all ending with 2026020900.0.0
27642026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all starting from 2026020900.0.0
27652026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all ending with 2026020900.0.0
27662026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/metrics/{metric_name} versions = all
27672026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/multicast-groups/by-ip/{address} versions = all ending with 2026010800.0.0
27682026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/address-lot versions = all
27692026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/networking/address-lot versions = all
27702026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
27712026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/address-lot/{address_lot} versions = all
27722026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
27732026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/allow-list versions = all
27742026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/networking/allow-list versions = all
27752026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/networking/bfd-disable versions = all
27762026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/networking/bfd-enable versions = all
27772026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bfd-status versions = all
27782026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/networking/bgp versions = all
27792026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp versions = all starting from 2026021301.0.0
27802026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp versions = all ending with 2026021301.0.0
27812026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/networking/bgp versions = all starting from 2026021301.0.0
27822026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/networking/bgp versions = all ending with 2026021301.0.0
27832026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp-announce-set versions = all
27842026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
27852026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
27862026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
27872026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp-exported versions = all starting from 2026021301.0.0
27882026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp-exported versions = all ending with 2026021301.0.0
27892026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp-imported versions = all starting from 2026021301.0.0
27902026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp-message-history versions = all
27912026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all ending with 2026021301.0.0
27922026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp-status versions = all starting from 2026021301.0.0
27932026-03-10T20:08:08.624ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp-status versions = from 2025121200.0.0 to 2026021301.0.0
27942026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/bgp-status versions = all ending with 2025121200.0.0
27952026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/inbound-icmp versions = all
27962026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/networking/inbound-icmp versions = all
27972026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/loopback-address versions = all
27982026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/networking/loopback-address versions = all
27992026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28002026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28012026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/switch-port-settings versions = all
28022026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/networking/switch-port-settings versions = all starting from 2026021301.0.0
28032026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/networking/switch-port-settings versions = all ending with 2026021301.0.0
28042026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all starting from 2026021301.0.0
28052026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all ending with 2026021301.0.0
28062026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/policy versions = all
28072026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/policy versions = all
28082026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/scim/tokens versions = all
28092026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/scim/tokens versions = all
28102026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/scim/tokens/{token_id} versions = all
28112026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/scim/tokens/{token_id} versions = all
28122026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/silo-quotas versions = all
28132026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/silos versions = all
28142026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/silos versions = all
28152026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/silos/{silo} versions = all
28162026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/silos/{silo} versions = all
28172026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all starting from 2026010100.0.0
28182026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all ending with 2026010100.0.0
28192026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/silos/{silo}/policy versions = all
28202026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28212026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/silos/{silo}/quotas versions = all
28222026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
28232026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/silos/{silo}/subnet-pools versions = all starting from 2026020600.0.0
28242026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools versions = all starting from 2026020900.0.0
28252026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools versions = from 2026012800.0.0 to 2026020900.0.0
28262026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools versions = from 2026011601.0.0 to 2026012800.0.0
28272026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools versions = all starting from 2026020900.0.0
28282026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools versions = from 2026012800.0.0 to 2026020900.0.0
28292026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools versions = from 2026012201.0.0 to 2026012800.0.0
28302026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools versions = from 2026011601.0.0 to 2026012201.0.0
28312026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/subnet-pools/{pool} versions = all starting from 2026020900.0.0
28322026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/subnet-pools/{pool} versions = from 2026011601.0.0 to 2026020900.0.0
28332026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools/{pool} versions = all starting from 2026020900.0.0
28342026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools/{pool} versions = from 2026012800.0.0 to 2026020900.0.0
28352026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools/{pool} versions = from 2026011601.0.0 to 2026012800.0.0
28362026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/subnet-pools/{pool} versions = all starting from 2026020900.0.0
28372026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/subnet-pools/{pool} versions = from 2026012800.0.0 to 2026020900.0.0
28382026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/subnet-pools/{pool} versions = from 2026011601.0.0 to 2026012800.0.0
28392026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools/{pool}/members versions = all starting from 2026020900.0.0
28402026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools/{pool}/members versions = from 2026012800.0.0 to 2026020900.0.0
28412026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools/{pool}/members versions = from 2026011601.0.0 to 2026012800.0.0
28422026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools/{pool}/members/add versions = all starting from 2026020900.0.0
28432026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools/{pool}/members/add versions = from 2026012800.0.0 to 2026020900.0.0
28442026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools/{pool}/members/add versions = from 2026012201.0.0 to 2026012800.0.0
28452026-03-10T20:08:08.625ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools/{pool}/members/add versions = from 2026011601.0.0 to 2026012201.0.0
28462026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools/{pool}/members/remove versions = all starting from 2026020900.0.0
28472026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools/{pool}/members/remove versions = from 2026011601.0.0 to 2026020900.0.0
28482026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools/{pool}/silos versions = all starting from 2026020900.0.0
28492026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools/{pool}/silos versions = from 2026011601.0.0 to 2026020900.0.0
28502026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools/{pool}/silos versions = all starting from 2026020900.0.0
28512026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/subnet-pools/{pool}/silos versions = from 2026011601.0.0 to 2026020900.0.0
28522026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/subnet-pools/{pool}/silos/{silo} versions = all starting from 2026020900.0.0
28532026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/subnet-pools/{pool}/silos/{silo} versions = from 2026011601.0.0 to 2026020900.0.0
28542026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/subnet-pools/{pool}/silos/{silo} versions = all starting from 2026020900.0.0
28552026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/subnet-pools/{pool}/silos/{silo} versions = from 2026011601.0.0 to 2026020900.0.0
28562026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools/{pool}/utilization versions = all starting from 2026020900.0.0
28572026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/subnet-pools/{pool}/utilization versions = from 2026011601.0.0 to 2026020900.0.0
28582026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/timeseries/query versions = all
28592026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/timeseries/schemas versions = all
28602026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/update/recovery-finish versions = all starting from 2026022500.0.0
28612026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/update/repositories versions = all
28622026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/update/repositories versions = all
28632026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/update/repositories/{system_version} versions = all
28642026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/update/status versions = all
28652026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/system/update/target-release versions = all
28662026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/update/trust-roots versions = all
28672026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/system/update/trust-roots versions = all
28682026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/system/update/trust-roots/{trust_root_id} versions = all
28692026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/update/trust-roots/{trust_root_id} versions = all
28702026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/users versions = all starting from 2026030200.0.0
28712026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/users versions = all ending with 2026030200.0.0
28722026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/users/{user_id} versions = all starting from 2026030200.0.0
28732026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/users/{user_id} versions = all ending with 2026030200.0.0
28742026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/users-builtin versions = all
28752026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/users-builtin/{user} versions = all
28762026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/utilization/silos versions = all
28772026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/system/utilization/silos/{silo} versions = all
28782026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/timeseries/query versions = all
28792026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/users versions = all starting from 2026030200.0.0
28802026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/users versions = all ending with 2026030200.0.0
28812026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/users/{user_id} versions = all starting from 2026030200.0.0
28822026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/users/{user_id} versions = all ending with 2026030200.0.0
28832026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/users/{user_id}/access-tokens versions = all
28842026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/users/{user_id}/logout versions = all
28852026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/users/{user_id}/sessions versions = all
28862026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/utilization versions = all
28872026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpc-firewall-rules versions = all
28882026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/vpc-firewall-rules versions = all
28892026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpc-router-routes versions = all
28902026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/vpc-router-routes versions = all
28912026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
28922026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpc-router-routes/{route} versions = all
28932026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/vpc-router-routes/{route} versions = all
28942026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpc-routers versions = all
28952026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/vpc-routers versions = all
28962026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/vpc-routers/{router} versions = all
28972026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpc-routers/{router} versions = all
28982026-03-10T20:08:08.626ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/vpc-routers/{router} versions = all
28992026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpc-subnets versions = all
29002026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/vpc-subnets versions = all
29012026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
29022026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpc-subnets/{subnet} versions = all
29032026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
29042026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all starting from 2026010300.0.0
29052026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all ending with 2026010300.0.0
29062026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpcs versions = all
29072026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/vpcs versions = all
29082026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/vpcs/{vpc} versions = all
29092026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/vpcs/{vpc} versions = all
29102026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/vpcs/{vpc} versions = all
29112026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/webhook-receivers versions = all
29122026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = PUT path = /v1/webhook-receivers/{receiver} versions = all
29132026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = GET path = /v1/webhook-secrets versions = all
29142026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = POST path = /v1/webhook-secrets versions = all
29152026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): registered endpoint local_addr = 127.0.0.1:46842 method = DELETE path = /v1/webhook-secrets/{secret_id} versions = all
29162026-03-10T20:08:08.627ZINFOtest_retry_backoff (dropshot_external): listening local_addr = 127.0.0.1:46842
29172026-03-10T20:08:08.627ZDEBGtest_retry_backoff (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:46842
29182026-03-10T20:08:08.657ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = / versions = all
29192026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /access versions = all
29202026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /assets/{path} versions = all
29212026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /device/auth versions = all
29222026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /device/confirm versions = all
29232026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /device/success versions = all
29242026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /device/token versions = all
29252026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /device/verify versions = all
29262026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /experimental/v1/probes versions = all starting from 2026010300.0.0
29272026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /experimental/v1/probes versions = all ending with 2026010300.0.0
29282026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /experimental/v1/probes versions = all starting from 2026010500.0.0
29292026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /experimental/v1/probes versions = all ending with 2026010500.0.0
29302026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /experimental/v1/probes/{probe} versions = all
29312026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /experimental/v1/probes/{probe} versions = all starting from 2026010300.0.0
29322026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /experimental/v1/probes/{probe} versions = all ending with 2026010300.0.0
29332026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /experimental/v1/system/support-bundles versions = all
29342026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /experimental/v1/system/support-bundles versions = all
29352026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /experimental/v1/system/support-bundles/{bundle_id} versions = all
29362026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /experimental/v1/system/support-bundles/{bundle_id} versions = all
29372026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /experimental/v1/system/support-bundles/{bundle_id} versions = all
29382026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /experimental/v1/system/support-bundles/{bundle_id}/download versions = all
29392026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = HEAD path = /experimental/v1/system/support-bundles/{bundle_id}/download versions = all
29402026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /experimental/v1/system/support-bundles/{bundle_id}/download/{file} versions = all
29412026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = HEAD path = /experimental/v1/system/support-bundles/{bundle_id}/download/{file} versions = all
29422026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /experimental/v1/system/support-bundles/{bundle_id}/index versions = all
29432026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /images versions = all
29442026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /login versions = all
29452026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /login/{silo_name}/local versions = all
29462026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
29472026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
29482026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
29492026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /lookup/{path} versions = all
29502026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /projects/{path} versions = all
29512026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /projects-new versions = all
29522026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /scim/v2/Groups versions = all
29532026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /scim/v2/Groups versions = all
29542026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /scim/v2/Groups/{group_id} versions = all
29552026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /scim/v2/Groups/{group_id} versions = all
29562026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PATCH path = /scim/v2/Groups/{group_id} versions = all
29572026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /scim/v2/Groups/{group_id} versions = all
29582026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /scim/v2/Users versions = all
29592026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /scim/v2/Users versions = all
29602026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /scim/v2/Users/{user_id} versions = all
29612026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /scim/v2/Users/{user_id} versions = all
29622026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PATCH path = /scim/v2/Users/{user_id} versions = all
29632026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /scim/v2/Users/{user_id} versions = all
29642026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /settings/{path} versions = all
29652026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /system/{path} versions = all
29662026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /utilization versions = all
29672026-03-10T20:08:08.658ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/affinity-groups versions = all
29682026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/affinity-groups versions = all
29692026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
29702026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
29712026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
29722026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
29732026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
29742026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
29752026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
29762026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/alert-classes versions = all
29772026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/alert-receivers versions = all
29782026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/alert-receivers/{receiver} versions = all
29792026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/alert-receivers/{receiver} versions = all
29802026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/alert-receivers/{receiver}/deliveries versions = all
29812026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/alert-receivers/{receiver}/probe versions = all
29822026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/alert-receivers/{receiver}/subscriptions versions = all
29832026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/alert-receivers/{receiver}/subscriptions/{subscription} versions = all
29842026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/alerts/{alert_id}/resend versions = all
29852026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/anti-affinity-groups versions = all
29862026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/anti-affinity-groups versions = all
29872026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
29882026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
29892026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
29902026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
29912026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
29922026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
29932026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
29942026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/auth-settings versions = all
29952026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/auth-settings versions = all
29962026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/certificates versions = all
29972026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/certificates versions = all
29982026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/certificates/{certificate} versions = all
29992026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/certificates/{certificate} versions = all
30002026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/disks versions = all starting from 2026013001.0.0
30012026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/disks versions = from 2025120300.0.0 to 2026013001.0.0
30022026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/disks versions = all ending with 2025120300.0.0
30032026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/disks versions = all starting from 2026013100.0.0
30042026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/disks versions = from 2026013001.0.0 to 2026013100.0.0
30052026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/disks versions = from 2025120300.0.0 to 2026013001.0.0
30062026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/disks versions = all ending with 2025120300.0.0
30072026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/disks/{disk} versions = all
30082026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/disks/{disk} versions = all starting from 2026013001.0.0
30092026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/disks/{disk} versions = from 2025120300.0.0 to 2026013001.0.0
30102026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/disks/{disk} versions = all ending with 2025120300.0.0
30112026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/disks/{disk}/bulk-write versions = all
30122026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
30132026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
30142026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/disks/{disk}/finalize versions = all
30152026-03-10T20:08:08.659ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/external-subnets versions = all starting from 2026011601.0.0
30162026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/external-subnets versions = all starting from 2026012201.0.0
30172026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/external-subnets versions = from 2026011601.0.0 to 2026012201.0.0
30182026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/external-subnets/{external_subnet} versions = all starting from 2026011601.0.0
30192026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/external-subnets/{external_subnet} versions = all starting from 2026011601.0.0
30202026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/external-subnets/{external_subnet} versions = all starting from 2026011601.0.0
30212026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/external-subnets/{external_subnet}/attach versions = all starting from 2026011601.0.0
30222026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/external-subnets/{external_subnet}/detach versions = all starting from 2026011601.0.0
30232026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/floating-ips versions = all
30242026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/floating-ips versions = all starting from 2026012200.0.0
30252026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/floating-ips versions = from 2026011600.0.0 to 2026012200.0.0
30262026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/floating-ips versions = from 2026010500.0.0 to 2026011600.0.0
30272026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/floating-ips versions = from 2025122300.0.0 to 2026010500.0.0
30282026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/floating-ips versions = all ending with 2025122300.0.0
30292026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
30302026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/floating-ips/{floating_ip} versions = all
30312026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
30322026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
30332026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
30342026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/groups versions = all starting from 2026030200.0.0
30352026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/groups versions = all ending with 2026030200.0.0
30362026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/groups/{group_id} versions = all starting from 2026030200.0.0
30372026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/groups/{group_id} versions = all ending with 2026030200.0.0
30382026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/images versions = all
30392026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/images versions = all
30402026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/images/{image} versions = all
30412026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/images/{image} versions = all
30422026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/images/{image}/demote versions = all
30432026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/images/{image}/promote versions = all
30442026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances versions = all
30452026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances versions = all starting from 2026013100.0.0
30462026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances versions = from 2026013001.0.0 to 2026013100.0.0
30472026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances versions = from 2026010800.0.0 to 2026013001.0.0
30482026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances versions = from 2026010500.0.0 to 2026010800.0.0
30492026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances versions = from 2026010300.0.0 to 2026010500.0.0
30502026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances versions = from 2025122300.0.0 to 2026010300.0.0
30512026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances versions = from 2025120300.0.0 to 2025122300.0.0
30522026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances versions = all ending with 2025120300.0.0
30532026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/instances/{instance} versions = all
30542026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance} versions = all
30552026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/instances/{instance} versions = all starting from 2026010800.0.0
30562026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/instances/{instance} versions = all ending with 2026010800.0.0
30572026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
30582026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
30592026-03-10T20:08:08.660ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/disks versions = all starting from 2026013001.0.0
30602026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/disks versions = from 2025120300.0.0 to 2026013001.0.0
30612026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/disks versions = all ending with 2025120300.0.0
30622026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/disks/attach versions = all starting from 2026013001.0.0
30632026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/disks/attach versions = from 2025120300.0.0 to 2026013001.0.0
30642026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/disks/attach versions = all ending with 2025120300.0.0
30652026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/disks/detach versions = all starting from 2026013001.0.0
30662026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/disks/detach versions = from 2025120300.0.0 to 2026013001.0.0
30672026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/disks/detach versions = all ending with 2025120300.0.0
30682026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/external-ips versions = all
30692026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all starting from 2026012300.0.0
30702026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all ending with 2026012300.0.0
30712026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all starting from 2026010500.0.0
30722026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = from 2025122300.0.0 to 2026010500.0.0
30732026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all ending with 2025122300.0.0
30742026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/external-subnets versions = all starting from 2026013000.0.0
30752026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/multicast-groups versions = all starting from 2026010800.0.0
30762026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/multicast-groups versions = all ending with 2026010800.0.0
30772026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/instances/{instance}/multicast-groups/{multicast_group} versions = all starting from 2026010800.0.0
30782026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/instances/{instance}/multicast-groups/{multicast_group} versions = all ending with 2026010800.0.0
30792026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/instances/{instance}/multicast-groups/{multicast_group} versions = all starting from 2026010800.0.0
30802026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/instances/{instance}/multicast-groups/{multicast_group} versions = all ending with 2026010800.0.0
30812026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/reboot versions = all
30822026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/serial-console versions = all
30832026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
30842026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
30852026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/start versions = all
30862026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/instances/{instance}/stop versions = all
30872026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/internet-gateway-ip-addresses versions = all
30882026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/internet-gateway-ip-addresses versions = all
30892026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
30902026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/internet-gateway-ip-pools versions = all
30912026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/internet-gateway-ip-pools versions = all
30922026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
30932026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/internet-gateways versions = all
30942026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/internet-gateways versions = all
30952026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
30962026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/internet-gateways/{gateway} versions = all
30972026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/ip-pools versions = all starting from 2026020900.0.0
30982026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/ip-pools versions = from 2026010100.0.0 to 2026020900.0.0
30992026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/ip-pools versions = all ending with 2026010100.0.0
31002026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/ip-pools/{pool} versions = all starting from 2026020900.0.0
31012026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/ip-pools/{pool} versions = from 2026010100.0.0 to 2026020900.0.0
31022026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/ip-pools/{pool} versions = all ending with 2026010100.0.0
31032026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/login/{silo_name}/local versions = all
31042026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/logout versions = all
31052026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/me versions = all starting from 2026030200.0.0
31062026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/me versions = all ending with 2026030200.0.0
31072026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/me/access-tokens versions = all
31082026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/me/access-tokens/{token_id} versions = all
31092026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/me/groups versions = all starting from 2026030200.0.0
31102026-03-10T20:08:08.661ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/me/groups versions = all ending with 2026030200.0.0
31112026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/me/ssh-keys versions = all
31122026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/me/ssh-keys versions = all
31132026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
31142026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
31152026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/metrics/{metric_name} versions = all
31162026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/multicast-groups versions = all starting from 2026010800.0.0
31172026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/multicast-groups versions = all ending with 2026010800.0.0
31182026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/multicast-groups versions = all ending with 2026010800.0.0
31192026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/multicast-groups/{multicast_group} versions = all ending with 2026010800.0.0
31202026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/multicast-groups/{multicast_group} versions = all starting from 2026010800.0.0
31212026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/multicast-groups/{multicast_group} versions = all ending with 2026010800.0.0
31222026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/multicast-groups/{multicast_group} versions = all ending with 2026010800.0.0
31232026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/multicast-groups/{multicast_group}/members versions = all starting from 2026010800.0.0
31242026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/multicast-groups/{multicast_group}/members versions = all ending with 2026010800.0.0
31252026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/multicast-groups/{multicast_group}/members versions = all ending with 2026010800.0.0
31262026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/multicast-groups/{multicast_group}/members/{instance} versions = all ending with 2026010800.0.0
31272026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/network-interfaces versions = all starting from 2026010300.0.0
31282026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/network-interfaces versions = all ending with 2026010300.0.0
31292026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/network-interfaces versions = all starting from 2026010300.0.0
31302026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/network-interfaces versions = all ending with 2026010300.0.0
31312026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/network-interfaces/{interface} versions = all
31322026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/network-interfaces/{interface} versions = all starting from 2026010300.0.0
31332026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/network-interfaces/{interface} versions = all ending with 2026010300.0.0
31342026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/network-interfaces/{interface} versions = all starting from 2026010300.0.0
31352026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/network-interfaces/{interface} versions = all ending with 2026010300.0.0
31362026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/ping versions = all
31372026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/policy versions = all
31382026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/policy versions = all
31392026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/projects versions = all
31402026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/projects versions = all
31412026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/projects/{project} versions = all
31422026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/projects/{project} versions = all
31432026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/projects/{project} versions = all
31442026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/projects/{project}/policy versions = all
31452026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/projects/{project}/policy versions = all
31462026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/snapshots versions = all
31472026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/snapshots versions = all
31482026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/snapshots/{snapshot} versions = all
31492026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/snapshots/{snapshot} versions = all
31502026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/subnet-pools versions = all starting from 2026020900.0.0
31512026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/subnet-pools versions = from 2026020600.0.0 to 2026020900.0.0
31522026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/subnet-pools/{pool} versions = all starting from 2026020900.0.0
31532026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/audit-log versions = all
31542026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/disks versions = all
31552026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
31562026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
31572026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/racks versions = all
31582026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
31592026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/racks/{rack_id}/membership versions = all starting from 2026012100.0.0
31602026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/hardware/racks/{rack_id}/membership/abort versions = all starting from 2026020200.0.0
31612026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/hardware/racks/{rack_id}/membership/add versions = all starting from 2026012100.0.0
31622026-03-10T20:08:08.662ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/sleds versions = all
31632026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/hardware/sleds versions = all ending with 2026021900.0.0
31642026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
31652026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
31662026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
31672026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
31682026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
31692026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/switch-port versions = all
31702026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
31712026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
31722026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
31732026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
31742026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
31752026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/switches versions = all
31762026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
31772026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/identity-providers versions = all
31782026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/identity-providers/local/users versions = all starting from 2026030200.0.0
31792026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/identity-providers/local/users versions = all ending with 2026030200.0.0
31802026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
31812026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
31822026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/identity-providers/saml versions = all
31832026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
31842026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools versions = all starting from 2026020900.0.0
31852026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools versions = all ending with 2026020900.0.0
31862026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools versions = all starting from 2026020900.0.0
31872026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools versions = all ending with 2026020900.0.0
31882026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/ip-pools/{pool} versions = all starting from 2026020900.0.0
31892026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/ip-pools/{pool} versions = all ending with 2026020900.0.0
31902026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools/{pool} versions = all starting from 2026020900.0.0
31912026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools/{pool} versions = all ending with 2026020900.0.0
31922026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/ip-pools/{pool} versions = all starting from 2026020900.0.0
31932026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/ip-pools/{pool} versions = all ending with 2026020900.0.0
31942026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all starting from 2026020900.0.0
31952026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all ending with 2026020900.0.0
31962026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all starting from 2026020900.0.0
31972026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all ending with 2026020900.0.0
31982026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all starting from 2026020900.0.0
31992026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all ending with 2026020900.0.0
32002026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all starting from 2026020900.0.0
32012026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all ending with 2026020900.0.0
32022026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all starting from 2026020900.0.0
32032026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all ending with 2026020900.0.0
32042026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all starting from 2026020900.0.0
32052026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all ending with 2026020900.0.0
32062026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all starting from 2026020900.0.0
32072026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all ending with 2026020900.0.0
32082026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all starting from 2026020900.0.0
32092026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all ending with 2026020900.0.0
32102026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools-service versions = all starting from 2026020900.0.0
32112026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools-service versions = all ending with 2026020900.0.0
32122026-03-10T20:08:08.663ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools-service/ranges versions = all starting from 2026020900.0.0
32132026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/ip-pools-service/ranges versions = all ending with 2026020900.0.0
32142026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all starting from 2026020900.0.0
32152026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all ending with 2026020900.0.0
32162026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all starting from 2026020900.0.0
32172026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all ending with 2026020900.0.0
32182026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/metrics/{metric_name} versions = all
32192026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/multicast-groups/by-ip/{address} versions = all ending with 2026010800.0.0
32202026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/address-lot versions = all
32212026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/networking/address-lot versions = all
32222026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
32232026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/address-lot/{address_lot} versions = all
32242026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
32252026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/allow-list versions = all
32262026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/networking/allow-list versions = all
32272026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/networking/bfd-disable versions = all
32282026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/networking/bfd-enable versions = all
32292026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bfd-status versions = all
32302026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/networking/bgp versions = all
32312026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp versions = all starting from 2026021301.0.0
32322026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp versions = all ending with 2026021301.0.0
32332026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/networking/bgp versions = all starting from 2026021301.0.0
32342026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/networking/bgp versions = all ending with 2026021301.0.0
32352026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp-announce-set versions = all
32362026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
32372026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
32382026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
32392026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp-exported versions = all starting from 2026021301.0.0
32402026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp-exported versions = all ending with 2026021301.0.0
32412026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp-imported versions = all starting from 2026021301.0.0
32422026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp-message-history versions = all
32432026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all ending with 2026021301.0.0
32442026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp-status versions = all starting from 2026021301.0.0
32452026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp-status versions = from 2025121200.0.0 to 2026021301.0.0
32462026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/bgp-status versions = all ending with 2025121200.0.0
32472026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/inbound-icmp versions = all
32482026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/networking/inbound-icmp versions = all
32492026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/loopback-address versions = all
32502026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/networking/loopback-address versions = all
32512026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
32522026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
32532026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/switch-port-settings versions = all
32542026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/networking/switch-port-settings versions = all starting from 2026021301.0.0
32552026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/networking/switch-port-settings versions = all ending with 2026021301.0.0
32562026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all starting from 2026021301.0.0
32572026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all ending with 2026021301.0.0
32582026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/policy versions = all
32592026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/policy versions = all
32602026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/scim/tokens versions = all
32612026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/scim/tokens versions = all
32622026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/scim/tokens/{token_id} versions = all
32632026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/scim/tokens/{token_id} versions = all
32642026-03-10T20:08:08.664ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/silo-quotas versions = all
32652026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/silos versions = all
32662026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/silos versions = all
32672026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/silos/{silo} versions = all
32682026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/silos/{silo} versions = all
32692026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all starting from 2026010100.0.0
32702026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all ending with 2026010100.0.0
32712026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/silos/{silo}/policy versions = all
32722026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/silos/{silo}/policy versions = all
32732026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/silos/{silo}/quotas versions = all
32742026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
32752026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/silos/{silo}/subnet-pools versions = all starting from 2026020600.0.0
32762026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools versions = all starting from 2026020900.0.0
32772026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools versions = from 2026012800.0.0 to 2026020900.0.0
32782026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools versions = from 2026011601.0.0 to 2026012800.0.0
32792026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools versions = all starting from 2026020900.0.0
32802026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools versions = from 2026012800.0.0 to 2026020900.0.0
32812026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools versions = from 2026012201.0.0 to 2026012800.0.0
32822026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools versions = from 2026011601.0.0 to 2026012201.0.0
32832026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/subnet-pools/{pool} versions = all starting from 2026020900.0.0
32842026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/subnet-pools/{pool} versions = from 2026011601.0.0 to 2026020900.0.0
32852026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools/{pool} versions = all starting from 2026020900.0.0
32862026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools/{pool} versions = from 2026012800.0.0 to 2026020900.0.0
32872026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools/{pool} versions = from 2026011601.0.0 to 2026012800.0.0
32882026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/subnet-pools/{pool} versions = all starting from 2026020900.0.0
32892026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/subnet-pools/{pool} versions = from 2026012800.0.0 to 2026020900.0.0
32902026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/subnet-pools/{pool} versions = from 2026011601.0.0 to 2026012800.0.0
32912026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools/{pool}/members versions = all starting from 2026020900.0.0
32922026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools/{pool}/members versions = from 2026012800.0.0 to 2026020900.0.0
32932026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools/{pool}/members versions = from 2026011601.0.0 to 2026012800.0.0
32942026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools/{pool}/members/add versions = all starting from 2026020900.0.0
32952026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools/{pool}/members/add versions = from 2026012800.0.0 to 2026020900.0.0
32962026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools/{pool}/members/add versions = from 2026012201.0.0 to 2026012800.0.0
32972026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools/{pool}/members/add versions = from 2026011601.0.0 to 2026012201.0.0
32982026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools/{pool}/members/remove versions = all starting from 2026020900.0.0
32992026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools/{pool}/members/remove versions = from 2026011601.0.0 to 2026020900.0.0
33002026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools/{pool}/silos versions = all starting from 2026020900.0.0
33012026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools/{pool}/silos versions = from 2026011601.0.0 to 2026020900.0.0
33022026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools/{pool}/silos versions = all starting from 2026020900.0.0
33032026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/subnet-pools/{pool}/silos versions = from 2026011601.0.0 to 2026020900.0.0
33042026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/subnet-pools/{pool}/silos/{silo} versions = all starting from 2026020900.0.0
33052026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/subnet-pools/{pool}/silos/{silo} versions = from 2026011601.0.0 to 2026020900.0.0
33062026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/subnet-pools/{pool}/silos/{silo} versions = all starting from 2026020900.0.0
33072026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/subnet-pools/{pool}/silos/{silo} versions = from 2026011601.0.0 to 2026020900.0.0
33082026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools/{pool}/utilization versions = all starting from 2026020900.0.0
33092026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/subnet-pools/{pool}/utilization versions = from 2026011601.0.0 to 2026020900.0.0
33102026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/timeseries/query versions = all
33112026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/timeseries/schemas versions = all
33122026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/update/recovery-finish versions = all starting from 2026022500.0.0
33132026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/update/repositories versions = all
33142026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/update/repositories versions = all
33152026-03-10T20:08:08.665ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/update/repositories/{system_version} versions = all
33162026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/update/status versions = all
33172026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/system/update/target-release versions = all
33182026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/update/trust-roots versions = all
33192026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/system/update/trust-roots versions = all
33202026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/system/update/trust-roots/{trust_root_id} versions = all
33212026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/update/trust-roots/{trust_root_id} versions = all
33222026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/users versions = all starting from 2026030200.0.0
33232026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/users versions = all ending with 2026030200.0.0
33242026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/users/{user_id} versions = all starting from 2026030200.0.0
33252026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/users/{user_id} versions = all ending with 2026030200.0.0
33262026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/users-builtin versions = all
33272026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/users-builtin/{user} versions = all
33282026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/utilization/silos versions = all
33292026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/system/utilization/silos/{silo} versions = all
33302026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/timeseries/query versions = all
33312026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/users versions = all starting from 2026030200.0.0
33322026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/users versions = all ending with 2026030200.0.0
33332026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/users/{user_id} versions = all starting from 2026030200.0.0
33342026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/users/{user_id} versions = all ending with 2026030200.0.0
33352026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/users/{user_id}/access-tokens versions = all
33362026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/users/{user_id}/logout versions = all
33372026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/users/{user_id}/sessions versions = all
33382026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/utilization versions = all
33392026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpc-firewall-rules versions = all
33402026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/vpc-firewall-rules versions = all
33412026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpc-router-routes versions = all
33422026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/vpc-router-routes versions = all
33432026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
33442026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpc-router-routes/{route} versions = all
33452026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/vpc-router-routes/{route} versions = all
33462026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpc-routers versions = all
33472026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/vpc-routers versions = all
33482026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/vpc-routers/{router} versions = all
33492026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpc-routers/{router} versions = all
33502026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/vpc-routers/{router} versions = all
33512026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpc-subnets versions = all
33522026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/vpc-subnets versions = all
33532026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
33542026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpc-subnets/{subnet} versions = all
33552026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
33562026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all starting from 2026010300.0.0
33572026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all ending with 2026010300.0.0
33582026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpcs versions = all
33592026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/vpcs versions = all
33602026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/vpcs/{vpc} versions = all
33612026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/vpcs/{vpc} versions = all
33622026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/vpcs/{vpc} versions = all
33632026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/webhook-receivers versions = all
33642026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = PUT path = /v1/webhook-receivers/{receiver} versions = all
33652026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = GET path = /v1/webhook-secrets versions = all
33662026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = POST path = /v1/webhook-secrets versions = all
33672026-03-10T20:08:08.666ZDEBGtest_retry_backoff (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49317 method = DELETE path = /v1/webhook-secrets/{secret_id} versions = all
33682026-03-10T20:08:08.667ZINFOtest_retry_backoff (dropshot_external_techport): listening local_addr = 127.0.0.1:49317
33692026-03-10T20:08:08.667ZDEBGtest_retry_backoff (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:49317
33702026-03-10T20:08:08.667ZDEBGtest_retry_backoff (nexus-producer-server): registered DTrace probes
33712026-03-10T20:08:08.667ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = 127.0.0.1:33574 method = GET path = /{producer_id} versions = all
33722026-03-10T20:08:08.667ZINFOtest_retry_backoff (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328 local_addr = 127.0.0.1:33574
33732026-03-10T20:08:08.667ZDEBGtest_retry_backoff (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:33574
33742026-03-10T20:08:08.667ZDEBGtest_retry_backoff (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:33574
33752026-03-10T20:08:08.667ZDEBGtest_retry_backoff (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:63060
33762026-03-10T20:08:08.667ZDEBGtest_retry_backoff (nexus-producer-server): starting producer registration task
33772026-03-10T20:08:08.667ZINFOtest_retry_backoff (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:33574 file = oximeter/producer/src/lib.rs:281 interval = 10s producer_id = 913233fe-92a8-4635-9572-183f495429c4
33782026-03-10T20:08:08.667ZDEBGtest_retry_backoff (init_with_steps): Running step start_oximeter
33792026-03-10T20:08:08.667ZDEBGtest_retry_backoff: Starting Oximeter
33802026-03-10T20:08:08.667ZDEBGtest_retry_backoff (oximeter): registered DTrace probes
33812026-03-10T20:08:08.667ZINFOtest_retry_backoff (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:264
33822026-03-10T20:08:08.667ZDEBGtest_retry_backoff (oximeter): creating ClickHouse client
33832026-03-10T20:08:08.667ZDEBGtest_retry_backoff (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
33842026-03-10T20:08:08.667ZDEBGtest_retry_backoff (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
33852026-03-10T20:08:08.667ZDEBGtest_retry_backoff (producer-registration-task): will register with Nexus at 127.0.0.1:63060
33862026-03-10T20:08:08.700ZDEBGtest_retry_backoff (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:63060/metrics/producers
33872026-03-10T20:08:08.701ZINFOtest_retry_backoff (dropshot_internal): accepted connection local_addr = 127.0.0.1:63060 remote_addr = 127.0.0.1:61695
33882026-03-10T20:08:08.702ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:61695 req_id = a4e38ddb-b297-44d4-90d0-9ef1ab870521 uri = /metrics/producers
33892026-03-10T20:08:08.702ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:61695 req_id = a4e38ddb-b297-44d4-90d0-9ef1ab870521 uri = /metrics/producers version = 1.0.0
33902026-03-10T20:08:08.703ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:61695 req_id = a4e38ddb-b297-44d4-90d0-9ef1ab870521 resource = Database type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
33912026-03-10T20:08:08.703ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:61695 req_id = a4e38ddb-b297-44d4-90d0-9ef1ab870521 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
33922026-03-10T20:08:08.704ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:61695 req_id = a4e38ddb-b297-44d4-90d0-9ef1ab870521 resource = Database result = Ok(()) type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
33932026-03-10T20:08:08.705ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = SELECT MAX(value) AS max_version FROM oximeter.version;
33942026-03-10T20:08:08.717ZWARNtest_retry_backoff (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:900 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
33952026-03-10T20:08:08.717ZDEBGtest_retry_backoff (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
33962026-03-10T20:08:08.717ZINFOtest_retry_backoff (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:800 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
33972026-03-10T20:08:08.717ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = SELECT MAX(value) AS max_version FROM oximeter.version;
33982026-03-10T20:08:08.720ZWARNtest_retry_backoff (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:900 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
33992026-03-10T20:08:08.720ZINFOtest_retry_backoff (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:805 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f version = 0
34002026-03-10T20:08:08.721ZINFOtest_retry_backoff (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:810 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
34012026-03-10T20:08:08.721ZDEBGtest_retry_backoff (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
34022026-03-10T20:08:08.722ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = CREATE DATABASE IF NOT EXISTS oximeter
34032026-03-10T20:08:08.723ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
34042026-03-10T20:08:08.726ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34052026-03-10T20:08:08.735ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34062026-03-10T20:08:08.755ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34072026-03-10T20:08:08.760ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34082026-03-10T20:08:08.762ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34092026-03-10T20:08:08.768ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34102026-03-10T20:08:08.769ZINFOtest_retry_backoff (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 66351 local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:61695 req_id = a4e38ddb-b297-44d4-90d0-9ef1ab870521 response_code = 503 uri = /metrics/producers
34112026-03-10T20:08:08.769ZDEBGtest_retry_backoff (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "a4e38ddb-b297-44d4-90d0-9ef1ab870521", "content-length": "133", "date": "Tue, 10 Mar 2026 20:08:08 GMT"} })
34122026-03-10T20:08:08.769ZWARNtest_retry_backoff (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 105.472534ms error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "a4e38ddb-b297-44d4-90d0-9ef1ab870521", "content-length": "133", "date": "Tue, 10 Mar 2026 20:08:08 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "a4e38ddb-b297-44d4-90d0-9ef1ab870521" } file = oximeter/producer/src/lib.rs:377
34132026-03-10T20:08:08.771ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34142026-03-10T20:08:08.774ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34152026-03-10T20:08:08.792ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34162026-03-10T20:08:08.804ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34172026-03-10T20:08:08.807ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34182026-03-10T20:08:08.819ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34192026-03-10T20:08:08.822ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34202026-03-10T20:08:08.832ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34212026-03-10T20:08:08.839ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34222026-03-10T20:08:08.841ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34232026-03-10T20:08:08.843ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34242026-03-10T20:08:08.847ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34252026-03-10T20:08:08.853ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34262026-03-10T20:08:08.855ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34272026-03-10T20:08:08.859ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34282026-03-10T20:08:08.863ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34292026-03-10T20:08:08.865ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34302026-03-10T20:08:08.867ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34312026-03-10T20:08:08.871ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34322026-03-10T20:08:08.881ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34332026-03-10T20:08:08.883ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34342026-03-10T20:08:08.885ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34352026-03-10T20:08:08.889ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34362026-03-10T20:08:08.902ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34372026-03-10T20:08:08.905ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34382026-03-10T20:08:08.907ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34392026-03-10T20:08:08.908ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34402026-03-10T20:08:08.910ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34412026-03-10T20:08:08.912ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34422026-03-10T20:08:08.913ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34432026-03-10T20:08:08.915ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34442026-03-10T20:08:08.917ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34452026-03-10T20:08:08.918ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34462026-03-10T20:08:08.920ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
34472026-03-10T20:08:08.922ZINFOtest_retry_backoff (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:831 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f version = 14
34482026-03-10T20:08:08.922ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = INSERT INTO oximeter.version (*) VALUES (14, now());
34492026-03-10T20:08:08.940ZDEBGtest_retry_backoff (producer-registration-task): will register with Nexus at 127.0.0.1:63060
34502026-03-10T20:08:09.021ZDEBGtest_retry_backoff (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:63060/metrics/producers
34512026-03-10T20:08:09.025ZDEBGtest_retry_backoff (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = f74e15c3-9ffe-4f99-9ad2-ea8b27d3b09c
34522026-03-10T20:08:09.026ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:34406 method = GET path = /info versions = all
34532026-03-10T20:08:09.026ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:34406 method = GET path = /producers versions = all
34542026-03-10T20:08:09.026ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:34406 method = DELETE path = /producers/{producer_id} versions = all
34552026-03-10T20:08:09.026ZDEBGtest_retry_backoff (dropshot): registered endpoint local_addr = [::1]:34406 method = GET path = /producers/{producer_id} versions = all
34562026-03-10T20:08:09.026ZINFOtest_retry_backoff (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328 local_addr = [::1]:34406
34572026-03-10T20:08:09.027ZDEBGtest_retry_backoff (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:34406
34582026-03-10T20:08:09.028ZDEBGtest_retry_backoff (oximeter): registered USDT probes
34592026-03-10T20:08:09.028ZDEBGtest_retry_backoff (oximeter): contacting nexus
34602026-03-10T20:08:09.028ZINFOtest_retry_backoff (dropshot_internal): accepted connection local_addr = 127.0.0.1:63060 remote_addr = 127.0.0.1:59511
34612026-03-10T20:08:09.398ZDEBGtest_retry_backoff (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:63060/metrics/collectors
34622026-03-10T20:08:09.398ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:59511 req_id = 2ebcd72b-0e84-4c24-bb9b-7e3b8d358283 uri = /metrics/producers
34632026-03-10T20:08:09.398ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:59511 req_id = 2ebcd72b-0e84-4c24-bb9b-7e3b8d358283 uri = /metrics/producers version = 1.0.0
34642026-03-10T20:08:09.399ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:59511 req_id = 2ebcd72b-0e84-4c24-bb9b-7e3b8d358283 resource = Database type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
34652026-03-10T20:08:09.399ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:59511 req_id = 2ebcd72b-0e84-4c24-bb9b-7e3b8d358283 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
34662026-03-10T20:08:09.399ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:59511 req_id = 2ebcd72b-0e84-4c24-bb9b-7e3b8d358283 resource = Database result = Ok(()) type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
34672026-03-10T20:08:09.401ZINFOtest_retry_backoff (dropshot_internal): accepted connection local_addr = 127.0.0.1:63060 remote_addr = 127.0.0.1:44916
34682026-03-10T20:08:09.403ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:44916 req_id = 2b855a30-ff56-4a31-85cd-9b9aae3ab62d uri = /metrics/collectors
34692026-03-10T20:08:09.403ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:44916 req_id = 2b855a30-ff56-4a31-85cd-9b9aae3ab62d uri = /metrics/collectors version = 1.0.0
34702026-03-10T20:08:09.403ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:44916 req_id = 2b855a30-ff56-4a31-85cd-9b9aae3ab62d resource = Database type = user_builtin uri = /metrics/collectors user_builtin_id = 001de000-05e4-4000-8000-000000000002
34712026-03-10T20:08:09.403ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:44916 req_id = 2b855a30-ff56-4a31-85cd-9b9aae3ab62d roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors user_builtin_id = 001de000-05e4-4000-8000-000000000002
34722026-03-10T20:08:09.404ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:44916 req_id = 2b855a30-ff56-4a31-85cd-9b9aae3ab62d resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors user_builtin_id = 001de000-05e4-4000-8000-000000000002
34732026-03-10T20:08:09.405ZINFOtest_retry_backoff (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 6965 local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:59511 req_id = 2ebcd72b-0e84-4c24-bb9b-7e3b8d358283 response_code = 503 uri = /metrics/producers
34742026-03-10T20:08:09.408ZDEBGtest_retry_backoff (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "2ebcd72b-0e84-4c24-bb9b-7e3b8d358283", "content-length": "133", "date": "Tue, 10 Mar 2026 20:08:09 GMT"} })
34752026-03-10T20:08:09.408ZWARNtest_retry_backoff (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 744.2297ms error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "2ebcd72b-0e84-4c24-bb9b-7e3b8d358283", "content-length": "133", "date": "Tue, 10 Mar 2026 20:08:09 GMT"}; value: Error { error_code: Some("ServiceNotAvailable"), message: "Service Unavailable", request_id: "2ebcd72b-0e84-4c24-bb9b-7e3b8d358283" } file = oximeter/producer/src/lib.rs:377
34762026-03-10T20:08:09.414ZINFOtest_retry_backoff (nexus): registered new oximeter metric collection server address = [::1]:34406 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
34772026-03-10T20:08:09.414ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 11300 local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:44916 req_id = 2b855a30-ff56-4a31-85cd-9b9aae3ab62d response_code = 204 uri = /metrics/collectors
34782026-03-10T20:08:09.415ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors", status: 204, headers: {"x-request-id": "2b855a30-ff56-4a31-85cd-9b9aae3ab62d", "date": "Tue, 10 Mar 2026 20:08:09 GMT"} })
34792026-03-10T20:08:09.415ZINFOtest_retry_backoff (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:427 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
34802026-03-10T20:08:09.415ZDEBGtest_retry_backoff (init_with_steps): Running step start_producer_server
34812026-03-10T20:08:09.415ZDEBGtest_retry_backoff: Starting test metric Producer Server
34822026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = Action
34832026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = AuditLog
34842026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = AnyActor
34852026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = AuthenticatedActor
34862026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = BlueprintConfig
34872026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = Database
34882026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = DnsConfig
34892026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = Fleet
34902026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = Inventory
34912026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = IpPoolList
34922026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = VpcList
34932026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = ConsoleSessionList
34942026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = DeviceAuthRequestList
34952026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = QuiesceState
34962026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = SiloCertificateList
34972026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = SiloGroupList
34982026-03-10T20:08:09.417ZINFOtest_retry_backoff: registering Oso class class = SiloIdentityProviderList
34992026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SiloUserList
35002026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SiloUserSessionList
35012026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SiloUserTokenList
35022026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SubnetPoolList
35032026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = UpdateTrustRootList
35042026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = TargetReleaseConfig
35052026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = AlertClassList
35062026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = ScimClientBearerTokenList
35072026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = MulticastGroupList
35082026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Project
35092026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Disk
35102026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Snapshot
35112026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = ProjectImage
35122026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = AffinityGroup
35132026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = AntiAffinityGroup
35142026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Instance
35152026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = IpPool
35162026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = InstanceNetworkInterface
35172026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Vpc
35182026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = VpcRouter
35192026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = InternetGateway
35202026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = InternetGatewayIpPool
35212026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = InternetGatewayIpAddress
35222026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = RouterRoute
35232026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = VpcSubnet
35242026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = FloatingIp
35252026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = ExternalSubnet
35262026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Image
35272026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SiloImage
35282026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = AddressLot
35292026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Blueprint
35302026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = LoopbackAddress
35312026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Certificate
35322026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = ConsoleSession
35332026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = DeviceAuthRequest
35342026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = DeviceAccessToken
35352026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = PhysicalDisk
35362026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Rack
35372026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SshKey
35382026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Silo
35392026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SiloUser
35402026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SiloGroup
35412026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SupportBundle
35422026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = IdentityProvider
35432026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SamlIdentityProvider
35442026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Sled
35452026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = SubnetPool
35462026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = TufRepo
35472026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = TufArtifact
35482026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = TufTrustRoot
35492026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Alert
35502026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = AlertReceiver
35512026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = WebhookSecret
35522026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Zpool
35532026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = Service
35542026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = UserBuiltin
35552026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = ScimClientBearerToken
35562026-03-10T20:08:09.418ZINFOtest_retry_backoff: registering Oso class class = MulticastGroup
35572026-03-10T20:08:09.418ZINFOtest_retry_backoff: full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations; grants project.admin on all projects)\n# - silo.limited-collaborator (grants project.limited-collaborator on all projects)\n# - silo.viewer (can read most resources within the Silo; grants project.viewer)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project, including networking)\n# - project.limited-collaborator (can manage compute resources, but not networking resources)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\n# Note that certain Actors may not ever have any roles assigned to them, like\n# SCIM Actors.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "limited-collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "limited-collaborator";\n\t"limited-collaborator" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif actor.is_user and silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "limited-collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t# Role hierarchy: admin > collaborator > limited-collaborator > viewer\n\t#\n\t# The "limited-collaborator" role can create/modify non-networking\n\t# resources (instances, disks, etc.) but cannot create/modify networking\n\t# infrastructure (VPCs, subnets, routers, internet gateways).\n\t# See nexus/authz-macros for InProjectLimited vs InProjectFull.\n\t"viewer" if "limited-collaborator";\n\t"limited-collaborator" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "limited-collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"limited-collaborator" if "limited-collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if actor.is_user and silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n}\n# We want to allow the user to modify the ssh key but disallow a SCIM IdP token\n# from doing the same.\nhas_permission(actor: AuthenticatedActor, "modify", ssh_key: SshKey)\n\tif actor.is_user and has_permission(actor, "modify", ssh_key.silo_user);\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource ScimClientBearerToken {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n # necessary to authenticate SCIM actors\n\t"read" if "external-authenticator" on "parent_fleet";\n\n\t# Silo-level roles grant privileges for SCIM client tokens.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges for SCIM client tokens.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", scim_client_bearer_token: ScimClientBearerToken)\n\tif scim_client_bearer_token.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: ScimClientBearerToken)\n\tif collection.silo.fleet = fleet;\n\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the quiesce state of a particular Nexus instance.\n#\n# These authz checks must not require the database. We grant this directly to\n# callers of the internal API.\nresource QuiesceState {\n\tpermissions = [ "read", "modify" ];\n}\nhas_permission(USER_INTERNAL_API: AuthenticatedActor, "read", _q: QuiesceState);\nhas_permission(\n USER_INTERNAL_API: AuthenticatedActor,\n "modify",\n _q: QuiesceState\n);\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/update/trust-roots" in the API\nresource UpdateTrustRootList {\n\tpermissions = [ "list_children", "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"list_children" if "viewer" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: UpdateTrustRootList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif actor.is_user and silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for accessing "/v1/multicast-groups" in the API\n# Groups are created when the first instance joins and deleted when the last leaves.\nresource MulticastGroupList {\n\tpermissions = [ "list_children" ];\n\n\trelations = { parent_fleet: Fleet };\n\n\t# Fleet Viewers can list multicast groups\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", multicast_group_list: MulticastGroupList)\n\tif multicast_group_list.fleet = fleet;\n\n# Any authenticated user can list multicast groups in their fleet.\n# This enables silo users to discover groups for attaching instances,\n# without requiring the Fleet::Viewer role.\nhas_permission(actor: AuthenticatedActor, "list_children", multicast_group_list: MulticastGroupList)\n\tif actor.is_user and silo in actor.silo and silo.fleet = multicast_group_list.fleet;\n\n# MulticastGroup is a fleet-level discovery resource.\n# Join/leave authorization is gated by Instance::Modify, not the group itself.\nresource MulticastGroup {\n\tpermissions = [ "read", "list_children" ];\n\trelations = { parent_fleet: Fleet };\n}\nhas_relation(fleet: Fleet, "parent_fleet", multicast_group: MulticastGroup)\n\tif multicast_group.fleet = fleet;\n\n# Any authenticated user can read multicast groups in their fleet\nhas_permission(actor: AuthenticatedActor, "read", multicast_group: MulticastGroup)\n\tif actor.is_user and silo in actor.silo and silo.fleet = multicast_group.fleet;\n\n# Any authenticated user can list members of multicast groups in their fleet\nhas_permission(actor: AuthenticatedActor, "list_children", multicast_group: MulticastGroup)\n\tif actor.is_user and silo in actor.silo and silo.fleet = multicast_group.fleet;\n\n# Describes the policy for reading and writing the audit log\nresource AuditLog {\n\tpermissions = [\n\t "list_children", # retrieve audit log\n\t "create_child", # create audit log entry\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\n\t# Fleet viewers can read the audit log\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\n# Any actor should be able to write to the audit log because we need to be able\n# to write to the log from any request, authenticated or not. Audit log writes\n# are always a byproduct of other operations: there are no endpoints that allow\n# the user to write to the log deliberately. Note we use AuthenticatedActor\n# because we don't really mean unauthenticated -- in the case of login\n# operations, we use the external authenticator actor that creates the session\n# to authorize the audit log write.\nhas_permission(_actor: AuthenticatedActor, "create_child", _audit_log: AuditLog);\n\nhas_relation(fleet: Fleet, "parent_fleet", audit_log: AuditLog)\n\tif audit_log.fleet = fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Allow silo admins to delete and list user sessions\nresource SiloUserSessionList {\n permissions = [ "modify", "list_children" ];\n relations = { parent_silo: Silo };\n\n # A silo admin can modify (e.g., delete) a user's sessions.\n "modify" if "admin" on "parent_silo";\n\n # A silo admin can list a user's sessions.\n "list_children" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", authn_list: SiloUserSessionList)\n if authn_list.silo_user.silo = silo;\n\n# give users 'modify' and 'list_children' on their own sessions\nhas_permission(actor: AuthenticatedActor, "modify", authn_list: SiloUserSessionList)\n if actor.equals_silo_user(authn_list.silo_user);\nhas_permission(actor: AuthenticatedActor, "list_children", authn_list: SiloUserSessionList)\n if actor.equals_silo_user(authn_list.silo_user);\n\n# Allow silo admins to delete and list user access tokens\nresource SiloUserTokenList {\n permissions = [ "modify", "list_children" ];\n relations = { parent_silo: Silo };\n\n # A silo admin can modify (e.g., delete) a user's tokens.\n "modify" if "admin" on "parent_silo";\n\n # A silo admin can list a user's tokens.\n "list_children" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", authn_list: SiloUserTokenList)\n if authn_list.silo_user.silo = silo;\n\n# give users 'modify' and 'list_children' on their own tokens\nhas_permission(actor: AuthenticatedActor, "modify", authn_list: SiloUserTokenList)\n if actor.equals_silo_user(authn_list.silo_user);\nhas_permission(actor: AuthenticatedActor, "list_children", authn_list: SiloUserTokenList)\n if actor.equals_silo_user(authn_list.silo_user);\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# Grant SCIM IdP actors the permissions they need on users.\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if actor.is_scim_idp and silo_user.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "create_child", silo_user_list: SiloUserList)\n\tif actor.is_scim_idp and silo_user_list.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "modify", silo_user: SiloUser)\n\tif actor.is_scim_idp and silo_user.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "list_children", silo_user_list: SiloUserList)\n if actor.is_scim_idp and silo_user_list.silo in actor.silo;\n\n# Describes the policy for creating and managing Silo groups (mostly intended\n# for API-managed groups)\nresource SiloGroupList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the groups in the\n\t# Silo) can see the groups in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's groups. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloGroupList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloGroupList)\n\tif collection.silo.fleet = fleet;\n\n# Grant SCIM IdP actors the permissions they need on groups.\nhas_permission(actor: AuthenticatedActor, "read", silo_group: SiloGroup)\n if actor.is_scim_idp and silo_group.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "create_child", silo_group_list: SiloGroupList)\n\tif actor.is_scim_idp and silo_group_list.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "modify", silo_group: SiloGroup)\n\tif actor.is_scim_idp and silo_group.silo in actor.silo;\nhas_permission(actor: AuthenticatedActor, "list_children", silo_group_list: SiloGroupList)\n if actor.is_scim_idp and silo_group_list.silo in actor.silo;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest)\n if actor.is_user;\nhas_permission(actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest)\n if actor.is_user;\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_alert_receiver: AlertReceiver };\n\n\t"read" if "read" on "parent_alert_receiver";\n\t"modify" if "modify" on "parent_alert_receiver";\n}\n\nhas_relation(rx: AlertReceiver, "parent_alert_receiver", secret: WebhookSecret)\n\tif secret.alert_receiver = rx;\n\nresource AlertClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: AlertClassList)\n\tif collection.fleet = fleet;\n\nresource ScimClientBearerTokenList {\n\tpermissions = [ "create_child", "list_children" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges for SCIM client tokens.\n # These are all admin because being able to create these tokens would allow\n # a user to grant themselves admin by modifying group membership through SCIM calls\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_silo";\n\n # Fleet-level roles also grant privileges for SCIM client tokens, for\n # configuration before silo admins are present.\n\t"create_child" if "admin" on "parent_fleet";\n\t"list_children" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", scim_client_bearer_token_list: ScimClientBearerTokenList)\n\tif scim_client_bearer_token_list.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: ScimClientBearerTokenList)\n\tif collection.silo.fleet = fleet;\n\n# VpcList is a synthetic resource for controlling VPC creation.\n# Unlike other project resources, VPC creation requires the full "collaborator"\n# role rather than "limited-collaborator", enforcing the networking restriction.\n# This allows organizations to restrict who can reconfigure the network topology\n# while still allowing users with limited-collaborator to work with compute\n# resources (instances, disks, etc.) within the existing network.\nresource VpcList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { containing_project: Project };\n\n\t"list_children" if "read" on "containing_project";\n\t"create_child" if "collaborator" on "containing_project";\n}\nhas_relation(project: Project, "containing_project", collection: VpcList)\n\tif collection.project = project;\n\n# Describes the policy for accessing "/v1/system/subnet-pools" in the API\nresource SubnetPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the Subnet Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list External Subnet Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", subnet_pool_list: SubnetPoolList)\n\tif subnet_pool_list.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource ExternalSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "limited-collaborator" on "containing_project";\n "create_child" if "limited-collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ExternalSubnet)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource SubnetPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SubnetPool)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource TufTrustRoot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufTrustRoot)\n if child.fleet = fleet;\n \n\n resource Alert {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Alert)\n if child.fleet = fleet;\n \n\n resource AlertReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AlertReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n \n\n
35582026-03-10T20:08:09.441ZINFOtest_retry_backoff (external client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:46842/v1/webhook-receivers
35592026-03-10T20:08:09.442ZINFOtest_retry_backoff (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:554
35602026-03-10T20:08:09.484ZINFOtest_retry_backoff (dropshot_internal): accepted connection local_addr = 127.0.0.1:63060 remote_addr = 127.0.0.1:36817
35612026-03-10T20:08:09.484ZINFOtest_retry_backoff (dropshot_external): accepted connection local_addr = 127.0.0.1:46842 remote_addr = 127.0.0.1:57042
35622026-03-10T20:08:09.485ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
35632026-03-10T20:08:09.485ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
35642026-03-10T20:08:09.487ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 664e904d-0201-4fea-8894-2f4fb57f233a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
35652026-03-10T20:08:09.488ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 664e904d-0201-4fea-8894-2f4fb57f233a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
35662026-03-10T20:08:09.488ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:36817 req_id = 3b2bab71-6563-4f61-9d06-5ed252b24709 uri = /metrics/producers
35672026-03-10T20:08:09.488ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:36817 req_id = 3b2bab71-6563-4f61-9d06-5ed252b24709 uri = /metrics/producers version = 1.0.0
35682026-03-10T20:08:09.488ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 664e904d-0201-4fea-8894-2f4fb57f233a resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
35692026-03-10T20:08:09.489ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 664e904d-0201-4fea-8894-2f4fb57f233a roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
35702026-03-10T20:08:09.489ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 664e904d-0201-4fea-8894-2f4fb57f233a resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
35712026-03-10T20:08:09.491ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:36817 req_id = 3b2bab71-6563-4f61-9d06-5ed252b24709 resource = Database type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
35722026-03-10T20:08:09.491ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:36817 req_id = 3b2bab71-6563-4f61-9d06-5ed252b24709 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
35732026-03-10T20:08:09.491ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:36817 req_id = 3b2bab71-6563-4f61-9d06-5ed252b24709 resource = Database result = Ok(()) type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
35742026-03-10T20:08:09.493ZTRACtest_retry_backoff (dropshot_external): incoming request local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 uri = /v1/webhook-receivers
35752026-03-10T20:08:09.493ZDEBGtest_retry_backoff (dropshot_external): determined request API version local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 uri = /v1/webhook-receivers version = 2026030200.0.0
35762026-03-10T20:08:09.493ZTRACtest_retry_backoff (dropshot_external): authn: trying Spoof local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 uri = /v1/webhook-receivers
35772026-03-10T20:08:09.493ZDEBGtest_retry_backoff (dropshot_external): looking up silo for user local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 silo_user_id = 001de000-05e4-4000-8000-000000004007 uri = /v1/webhook-receivers
35782026-03-10T20:08:09.494ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
35792026-03-10T20:08:09.494ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
35802026-03-10T20:08:09.495ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
35812026-03-10T20:08:09.499ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 12868 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 664e904d-0201-4fea-8894-2f4fb57f233a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
35822026-03-10T20:08:09.500ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
35832026-03-10T20:08:09.500ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
35842026-03-10T20:08:09.500ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
35852026-03-10T20:08:09.506ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "664e904d-0201-4fea-8894-2f4fb57f233a", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:09 GMT"} })
35862026-03-10T20:08:09.506ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 0 n_pruned_tasks = 0
35872026-03-10T20:08:09.514ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007 (silo_user), lookup_type: ById(001de000-05e4-4000-8000-000000004007) }
35882026-03-10T20:08:09.514ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
35892026-03-10T20:08:09.514ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
35902026-03-10T20:08:09.514ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
35912026-03-10T20:08:09.515ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
35922026-03-10T20:08:09.518ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35932026-03-10T20:08:09.518ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
35942026-03-10T20:08:09.518ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
35952026-03-10T20:08:09.518ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
35962026-03-10T20:08:09.520ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
35972026-03-10T20:08:09.523ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007 (silo_user), lookup_type: ById(001de000-05e4-4000-8000-000000004007) } result = Ok(())
35982026-03-10T20:08:09.523ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
35992026-03-10T20:08:09.523ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
36002026-03-10T20:08:09.524ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
36012026-03-10T20:08:09.525ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }
36022026-03-10T20:08:09.525ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
36032026-03-10T20:08:09.525ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
36042026-03-10T20:08:09.525ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
36052026-03-10T20:08:09.526ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
36062026-03-10T20:08:09.527ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36072026-03-10T20:08:09.527ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
36082026-03-10T20:08:09.527ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
36092026-03-10T20:08:09.528ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
36102026-03-10T20:08:09.529ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
36112026-03-10T20:08:09.531ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) } result = Ok(())
36122026-03-10T20:08:09.531ZTRACtest_retry_backoff (dropshot_external): authn result: Ok(Context { kind: Authenticated(Details { actor: Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }, device_token_expiration: None, credential_id: None }, Some(SiloAuthnPolicy { mapped_fleet_roles: {} })), schemes_tried: [Spoof] }) local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 uri = /v1/webhook-receivers
36132026-03-10T20:08:09.531ZTRACtest_retry_backoff (dropshot_external): authorize begin action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = AuditLog silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36142026-03-10T20:08:09.531ZTRACtest_retry_backoff (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. } authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36152026-03-10T20:08:09.531ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36162026-03-10T20:08:09.531ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36172026-03-10T20:08:09.532ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36182026-03-10T20:08:09.536ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36192026-03-10T20:08:09.537ZDEBGtest_retry_backoff (dropshot_external): authorize result action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = AuditLog result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36202026-03-10T20:08:09.537ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36212026-03-10T20:08:09.537ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36222026-03-10T20:08:09.538ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36232026-03-10T20:08:09.538ZINFOtest_retry_backoff (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
36242026-03-10T20:08:09.539ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 50599 local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:36817 req_id = 3b2bab71-6563-4f61-9d06-5ed252b24709 response_code = 201 uri = /metrics/producers
36252026-03-10T20:08:09.543ZINFOtest_retry_backoff (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 217d854a-1530-402d-b94f-a3562c53f830
36262026-03-10T20:08:09.543ZDEBGtest_retry_backoff (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 217d854a-1530-402d-b94f-a3562c53f830 time_started = 2026-03-10 20:08:08.140360 UTC
36272026-03-10T20:08:09.543ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.652652094s iteration = 3 status = {"collection_id":"217d854a-1530-402d-b94f-a3562c53f830","time_done":"2026-03-10 20:08:08.477301 UTC","time_started":"2026-03-10 20:08:08.140360 UTC"}
36282026-03-10T20:08:09.543ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = inventory_loader iteration = 2 reason = Dependency
36292026-03-10T20:08:09.543ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Inventory
36302026-03-10T20:08:09.543ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = inventory_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36312026-03-10T20:08:09.543ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database
36322026-03-10T20:08:09.543ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {} }
36332026-03-10T20:08:09.543ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database result = Ok(())
36342026-03-10T20:08:09.545ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36352026-03-10T20:08:09.549ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Inventory result = Ok(())
36362026-03-10T20:08:09.549ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database
36372026-03-10T20:08:09.549ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {} }
36382026-03-10T20:08:09.549ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database result = Ok(())
36392026-03-10T20:08:09.551ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database
36402026-03-10T20:08:09.551ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {} }
36412026-03-10T20:08:09.551ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database result = Ok(())
36422026-03-10T20:08:09.616ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = inventory_loader elapsed = 73.18763ms iteration = 2 status = {"Loaded":{"collection_id":"217d854a-1530-402d-b94f-a3562c53f830","time_loaded":"2026-03-10T20:08:09.543438369Z","time_started":"2026-03-10T20:08:08.140360Z"}}
36432026-03-10T20:08:09.617ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
36442026-03-10T20:08:09.617ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 20.155\u{b5}s iteration = 2 status = {"error":"task disabled"}
36452026-03-10T20:08:09.617ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = blueprint_planner iteration = 4 reason = Dependency
36462026-03-10T20:08:09.617ZDEBGtest_retry_backoff (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
36472026-03-10T20:08:09.617ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 63.206\u{b5}s iteration = 4 status = "Disabled"
36482026-03-10T20:08:09.617ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = multicast_reconciler iteration = 2 reason = Dependency
36492026-03-10T20:08:09.617ZTRACtest_retry_backoff (BackgroundTasks): multicast group reconciler activating background_task = multicast_reconciler
36502026-03-10T20:08:09.617ZTRACtest_retry_backoff (BackgroundTasks): starting multicast reconciliation pass background_task = multicast_reconciler
36512026-03-10T20:08:09.617ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
36522026-03-10T20:08:09.617ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
36532026-03-10T20:08:09.617ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet
36542026-03-10T20:08:09.617ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_rendezvous resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36552026-03-10T20:08:09.617ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
36562026-03-10T20:08:09.617ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
36572026-03-10T20:08:09.618ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
36582026-03-10T20:08:09.621ZTRACtest_retry_backoff (dropshot_external): authorize begin action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Fleet silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36592026-03-10T20:08:09.621ZTRACtest_retry_backoff (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. } authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36602026-03-10T20:08:09.621ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36612026-03-10T20:08:09.621ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36622026-03-10T20:08:09.621ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36632026-03-10T20:08:09.623ZTRACtest_retry_backoff (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: b1 a1 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:16121 req_id = 83681c73-649c-47fb-bc99-623976131ccd
36642026-03-10T20:08:09.623ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45473,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:16121 req_id = 83681c73-649c-47fb-bc99-623976131ccd
36652026-03-10T20:08:09.623ZDEBGtest_retry_backoff (store): query key key = _dendrite._tcp
36662026-03-10T20:08:09.623ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36672026-03-10T20:08:09.623ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:16121 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = 83681c73-649c-47fb-bc99-623976131ccd
36682026-03-10T20:08:09.624ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36692026-03-10T20:08:09.627ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
36702026-03-10T20:08:09.627ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet
36712026-03-10T20:08:09.627ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_rendezvous resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36722026-03-10T20:08:09.627ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
36732026-03-10T20:08:09.627ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
36742026-03-10T20:08:09.628ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
36752026-03-10T20:08:09.628ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36762026-03-10T20:08:09.631ZDEBGtest_retry_backoff (dropshot_external): authorize result action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Fleet result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36772026-03-10T20:08:09.631ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36782026-03-10T20:08:09.631ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36792026-03-10T20:08:09.631ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
36802026-03-10T20:08:09.632ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6309, tv_nsec: 492649027 } })
36812026-03-10T20:08:09.632ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
36822026-03-10T20:08:09.633ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36832026-03-10T20:08:09.636ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
36842026-03-10T20:08:09.636ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
36852026-03-10T20:08:09.636ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
36862026-03-10T20:08:09.636ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
36872026-03-10T20:08:09.639ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2d 10 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e -...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49838 req_id = 9282ff76-bcd2-421e-b2e2-b40b2af96fdc
36882026-03-10T20:08:09.639ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11536,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:49838 req_id = 9282ff76-bcd2-421e-b2e2-b40b2af96fdc
36892026-03-10T20:08:09.639ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36902026-03-10T20:08:09.639ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49838 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 9282ff76-bcd2-421e-b2e2-b40b2af96fdc
36912026-03-10T20:08:09.675ZTRACtest_retry_backoff (DpdClient): client request background_task = multicast_reconciler body = None method = GET uri = http://[::1]:41467/switch/identifiers
36922026-03-10T20:08:09.678ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet
36932026-03-10T20:08:09.678ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_rendezvous resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36942026-03-10T20:08:09.678ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
36952026-03-10T20:08:09.678ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
36962026-03-10T20:08:09.678ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
36972026-03-10T20:08:09.681ZTRACtest_retry_backoff (DpdClient): client response background_task = multicast_reconciler result = Ok(Response { url: "http://[::1]:41467/switch/identifiers", status: 200, headers: {"content-type": "application/json", "x-request-id": "4be72253-eb80-495b-9ea6-61c0f4fbefe1", "content-length": "717", "date": "Tue, 10 Mar 2026 20:08:09 GMT"} })
36982026-03-10T20:08:09.681ZTRACtest_retry_backoff (BackgroundTasks): searching for multicast groups background_task = multicast_reconciler state = Creating
36992026-03-10T20:08:09.681ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database
37002026-03-10T20:08:09.682ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = multicast_reconciler roles = RoleSet { roles: {} }
37012026-03-10T20:08:09.682ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database result = Ok(())
37022026-03-10T20:08:09.683ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37032026-03-10T20:08:09.686ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
37042026-03-10T20:08:09.686ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet
37052026-03-10T20:08:09.686ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_rendezvous resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37062026-03-10T20:08:09.686ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
37072026-03-10T20:08:09.686ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
37082026-03-10T20:08:09.686ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
37092026-03-10T20:08:09.695ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37102026-03-10T20:08:09.698ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
37112026-03-10T20:08:09.698ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
37122026-03-10T20:08:09.698ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
37132026-03-10T20:08:09.699ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
37142026-03-10T20:08:09.706ZINFOtest_retry_backoff (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous num_already_exist = 0 num_inserted = 0 num_not_in_inventory = 0
37152026-03-10T20:08:09.706ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet
37162026-03-10T20:08:09.706ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_rendezvous resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37172026-03-10T20:08:09.706ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
37182026-03-10T20:08:09.706ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
37192026-03-10T20:08:09.707ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
37202026-03-10T20:08:09.709ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37212026-03-10T20:08:09.712ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
37222026-03-10T20:08:09.712ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet
37232026-03-10T20:08:09.712ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_rendezvous resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37242026-03-10T20:08:09.712ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
37252026-03-10T20:08:09.712ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
37262026-03-10T20:08:09.712ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
37272026-03-10T20:08:09.714ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37282026-03-10T20:08:09.717ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
37292026-03-10T20:08:09.717ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
37302026-03-10T20:08:09.717ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
37312026-03-10T20:08:09.718ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
37322026-03-10T20:08:09.722ZINFOtest_retry_backoff (BackgroundTasks): all LocalStorage datasets reconciled background_task = blueprint_rendezvous num_already_exist = 0 num_already_tombstoned = 0 num_inserted = 0 num_not_in_inventory = 0 num_tombstoned = 0
37332026-03-10T20:08:09.722ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet
37342026-03-10T20:08:09.722ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_rendezvous resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37352026-03-10T20:08:09.722ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
37362026-03-10T20:08:09.722ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
37372026-03-10T20:08:09.722ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
37382026-03-10T20:08:09.724ZDEBGtest_retry_backoff (dropshot_external): added exact subscription to alert receiver authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 subscription = Some(AlertRxSubscription { rx_id: 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver), class: TestFoo, glob: None, time_created: 2026-03-10T20:08:09.679831Z }) type = silo_user uri = /v1/webhook-receivers
37392026-03-10T20:08:09.724ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37402026-03-10T20:08:09.727ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
37412026-03-10T20:08:09.727ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet
37422026-03-10T20:08:09.727ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = blueprint_rendezvous resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37432026-03-10T20:08:09.727ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
37442026-03-10T20:08:09.727ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
37452026-03-10T20:08:09.728ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
37462026-03-10T20:08:09.729ZTRACtest_retry_backoff (dropshot_external): authorize begin action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = AuditLog silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
37472026-03-10T20:08:09.729ZTRACtest_retry_backoff (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. } authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
37482026-03-10T20:08:09.729ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
37492026-03-10T20:08:09.729ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
37502026-03-10T20:08:09.730ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
37512026-03-10T20:08:09.730ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37522026-03-10T20:08:09.733ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
37532026-03-10T20:08:09.733ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database
37542026-03-10T20:08:09.734ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
37552026-03-10T20:08:09.734ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
37562026-03-10T20:08:09.734ZTRACtest_retry_backoff (BackgroundTasks): found multicast groups background_task = multicast_reconciler count = 0 state = Creating
37572026-03-10T20:08:09.734ZTRACtest_retry_backoff (BackgroundTasks): reconciling member state changes background_task = multicast_reconciler
37582026-03-10T20:08:09.734ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database
37592026-03-10T20:08:09.734ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = multicast_reconciler roles = RoleSet { roles: {} }
37602026-03-10T20:08:09.735ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database result = Ok(())
37612026-03-10T20:08:09.736ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
37622026-03-10T20:08:09.737ZDEBGtest_retry_backoff (dropshot_external): authorize result action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = AuditLog result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
37632026-03-10T20:08:09.737ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
37642026-03-10T20:08:09.737ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
37652026-03-10T20:08:09.738ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/webhook-receivers
37662026-03-10T20:08:09.738ZINFOtest_retry_backoff (BackgroundTasks): all LocalStorageUnencrypted datasets reconciled background_task = blueprint_rendezvous num_already_exist = 0 num_already_tombstoned = 0 num_inserted = 0 num_not_in_inventory = 0 num_tombstoned = 0
37672026-03-10T20:08:09.738ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 121.217523ms iteration = 2 status = {"blueprint_id":"6e43be5e-eba9-4a79-84a8-7f9521f2c946","inventory_collection_id":"217d854a-1530-402d-b94f-a3562c53f830","stats":{"crucible_dataset":{"num_already_exist":0,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":0,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0},"local_storage_dataset":{"num_already_exist":0,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0},"local_storage_unencrypted_dataset":{"num_already_exist":0,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
37682026-03-10T20:08:09.778ZDEBGtest_retry_backoff (BackgroundTasks): member state reconciliation completed background_task = multicast_reconciler members_processed = 0
37692026-03-10T20:08:09.778ZTRACtest_retry_backoff (BackgroundTasks): cleaning up deleted multicast members background_task = multicast_reconciler
37702026-03-10T20:08:09.778ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database
37712026-03-10T20:08:09.778ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = multicast_reconciler roles = RoleSet { roles: {} }
37722026-03-10T20:08:09.778ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database result = Ok(())
37732026-03-10T20:08:09.782ZINFOtest_retry_backoff (dropshot_external): request completed latency_us = 289259 local_addr = 127.0.0.1:46842 method = POST remote_addr = 127.0.0.1:57042 req_id = 8a08cc90-bea2-43a8-a189-e1b4a97c0ce3 response_code = 201 uri = /v1/webhook-receivers
37742026-03-10T20:08:09.782ZINFOtest_retry_backoff (external client test context): client received response status = 201
37752026-03-10T20:08:09.784ZTRACtest_retry_backoff: authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) resource = Database
37762026-03-10T20:08:09.784ZDEBGtest_retry_backoff: roles roles = RoleSet { roles: {} }
37772026-03-10T20:08:09.785ZDEBGtest_retry_backoff: authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) resource = Database result = Ok(())
37782026-03-10T20:08:09.818ZDEBGtest_retry_backoff: published alert alert_class = test.foo alert_id = 5b38ce09-0fe3-4310-a5b1-e7643a39c1de (alert) time_created = 2026-03-10T20:08:09.808622Z
37792026-03-10T20:08:09.818ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/alert_dispatcher
37802026-03-10T20:08:09.818ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = alert_dispatcher iteration = 2 reason = Signaled
37812026-03-10T20:08:09.818ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database
37822026-03-10T20:08:09.818ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
37832026-03-10T20:08:09.819ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database result = Ok(())
37842026-03-10T20:08:09.828ZINFOtest_retry_backoff (dropshot_lockstep): accepted connection local_addr = 127.0.0.1:50070 remote_addr = 127.0.0.1:43636
37852026-03-10T20:08:09.829ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 144373b0-7b2b-4248-b7fd-392db8083ba3 uri = /bgtasks/view/alert_dispatcher
37862026-03-10T20:08:09.829ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 144373b0-7b2b-4248-b7fd-392db8083ba3 resource = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
37872026-03-10T20:08:09.829ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 144373b0-7b2b-4248-b7fd-392db8083ba3 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
37882026-03-10T20:08:09.829ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 144373b0-7b2b-4248-b7fd-392db8083ba3 resource = Database type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
37892026-03-10T20:08:09.829ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 144373b0-7b2b-4248-b7fd-392db8083ba3 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
37902026-03-10T20:08:09.830ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 144373b0-7b2b-4248-b7fd-392db8083ba3 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
37912026-03-10T20:08:09.832ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 144373b0-7b2b-4248-b7fd-392db8083ba3 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
37922026-03-10T20:08:09.837ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 144373b0-7b2b-4248-b7fd-392db8083ba3 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
37932026-03-10T20:08:09.837ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 8703 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 144373b0-7b2b-4248-b7fd-392db8083ba3 response_code = 200 uri = /bgtasks/view/alert_dispatcher
37942026-03-10T20:08:09.837ZDEBGtest_retry_backoff (BackgroundTasks): multicast group member complete deletion finished background_task = multicast_reconciler left_and_time_deleted_members_deleted = 0
37952026-03-10T20:08:09.837ZTRACtest_retry_backoff (BackgroundTasks): checking for empty multicast groups to implicitly delete background_task = multicast_reconciler
37962026-03-10T20:08:09.837ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database
37972026-03-10T20:08:09.838ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = multicast_reconciler roles = RoleSet { roles: {} }
37982026-03-10T20:08:09.838ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database result = Ok(())
37992026-03-10T20:08:09.839ZTRACtest_retry_backoff (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = alert_dispatcher
38002026-03-10T20:08:09.839ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database
38012026-03-10T20:08:09.839ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
38022026-03-10T20:08:09.840ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database result = Ok(())
38032026-03-10T20:08:09.841ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
38042026-03-10T20:08:09.841ZINFOtest_retry_backoff (lockstep client test context): waiting for alert_dispatcher to go idle
38052026-03-10T20:08:09.842ZTRACtest_retry_backoff (BackgroundTasks): dispatching webhook event... alert_class = test.foo alert_id = 5b38ce09-0fe3-4310-a5b1-e7643a39c1de (alert) background_task = alert_dispatcher
38062026-03-10T20:08:09.842ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database
38072026-03-10T20:08:09.842ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
38082026-03-10T20:08:09.843ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database result = Ok(())
38092026-03-10T20:08:09.894ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/alert_dispatcher
38102026-03-10T20:08:09.896ZTRACtest_retry_backoff (BackgroundTasks): searching for multicast groups background_task = multicast_reconciler state = Active
38112026-03-10T20:08:09.896ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database
38122026-03-10T20:08:09.896ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = multicast_reconciler roles = RoleSet { roles: {} }
38132026-03-10T20:08:09.897ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database result = Ok(())
38142026-03-10T20:08:09.898ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 49c5c43c-f7c7-4173-8a4b-f44fe657e16e uri = /bgtasks/view/alert_dispatcher
38152026-03-10T20:08:09.898ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 49c5c43c-f7c7-4173-8a4b-f44fe657e16e resource = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38162026-03-10T20:08:09.898ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 49c5c43c-f7c7-4173-8a4b-f44fe657e16e resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38172026-03-10T20:08:09.898ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 49c5c43c-f7c7-4173-8a4b-f44fe657e16e resource = Database type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38182026-03-10T20:08:09.898ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 49c5c43c-f7c7-4173-8a4b-f44fe657e16e roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38192026-03-10T20:08:09.898ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 49c5c43c-f7c7-4173-8a4b-f44fe657e16e resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38202026-03-10T20:08:09.900ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 49c5c43c-f7c7-4173-8a4b-f44fe657e16e roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38212026-03-10T20:08:09.904ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 49c5c43c-f7c7-4173-8a4b-f44fe657e16e resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38222026-03-10T20:08:09.904ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 6439 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 49c5c43c-f7c7-4173-8a4b-f44fe657e16e response_code = 200 uri = /bgtasks/view/alert_dispatcher
38232026-03-10T20:08:09.904ZTRACtest_retry_backoff (BackgroundTasks): found multicast groups background_task = multicast_reconciler count = 0 state = Active
38242026-03-10T20:08:09.904ZTRACtest_retry_backoff (BackgroundTasks): searching for multicast groups background_task = multicast_reconciler state = Deleting
38252026-03-10T20:08:09.904ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database
38262026-03-10T20:08:09.904ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = multicast_reconciler roles = RoleSet { roles: {} }
38272026-03-10T20:08:09.905ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = multicast_reconciler resource = Database result = Ok(())
38282026-03-10T20:08:09.906ZTRACtest_retry_backoff (BackgroundTasks): alert receiver is subscribed to event alert_class = test.foo alert_id = 5b38ce09-0fe3-4310-a5b1-e7643a39c1de (alert) background_task = alert_dispatcher glob = None rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver) rx_name = my-great-webhook
38292026-03-10T20:08:09.906ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database
38302026-03-10T20:08:09.906ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
38312026-03-10T20:08:09.906ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database result = Ok(())
38322026-03-10T20:08:09.907ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
38332026-03-10T20:08:09.907ZINFOtest_retry_backoff (lockstep client test context): waiting for alert_dispatcher to go idle
38342026-03-10T20:08:09.952ZTRACtest_retry_backoff (BackgroundTasks): found multicast groups background_task = multicast_reconciler count = 0 state = Deleting
38352026-03-10T20:08:09.952ZTRACtest_retry_backoff (BackgroundTasks): multicast RPW reconciliation cycle completed active_groups_verified = 0 background_task = multicast_reconciler error_count = 0 external_groups_created = 0 external_groups_deleted = 0 member_lifecycle_transitions = 0 orphaned_member_cleanup = 0 total_dpd_operations = 0
38362026-03-10T20:08:09.952ZTRACtest_retry_backoff (BackgroundTasks): multicast RPW reconciliation pass completed - dataplane consistent background_task = multicast_reconciler
38372026-03-10T20:08:09.952ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = multicast_reconciler elapsed = 335.573109ms iteration = 2 status = {"disabled":false,"empty_groups_marked":0,"errors":[],"groups_created":0,"groups_deleted":0,"groups_verified":0,"members_deleted":0,"members_processed":0}
38382026-03-10T20:08:09.959ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/alert_dispatcher
38392026-03-10T20:08:09.959ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f649da79-8608-4678-9d95-bee0f7e6a652 uri = /bgtasks/view/alert_dispatcher
38402026-03-10T20:08:09.959ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f649da79-8608-4678-9d95-bee0f7e6a652 resource = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38412026-03-10T20:08:09.959ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f649da79-8608-4678-9d95-bee0f7e6a652 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38422026-03-10T20:08:09.959ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f649da79-8608-4678-9d95-bee0f7e6a652 resource = Database type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38432026-03-10T20:08:09.959ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f649da79-8608-4678-9d95-bee0f7e6a652 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38442026-03-10T20:08:09.960ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f649da79-8608-4678-9d95-bee0f7e6a652 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38452026-03-10T20:08:09.962ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f649da79-8608-4678-9d95-bee0f7e6a652 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38462026-03-10T20:08:09.966ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f649da79-8608-4678-9d95-bee0f7e6a652 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38472026-03-10T20:08:09.966ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 6696 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f649da79-8608-4678-9d95-bee0f7e6a652 response_code = 200 uri = /bgtasks/view/alert_dispatcher
38482026-03-10T20:08:09.966ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
38492026-03-10T20:08:09.966ZINFOtest_retry_backoff (lockstep client test context): waiting for alert_dispatcher to go idle
38502026-03-10T20:08:09.990ZDEBGtest_retry_backoff (BackgroundTasks): dispatched webhook event alert_class = test.foo alert_id = 5b38ce09-0fe3-4310-a5b1-e7643a39c1de (alert) background_task = alert_dispatcher num_dispatched = 1 num_subscribed = 1
38512026-03-10T20:08:09.990ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database
38522026-03-10T20:08:09.990ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
38532026-03-10T20:08:09.991ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database result = Ok(())
38542026-03-10T20:08:09.998ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database
38552026-03-10T20:08:09.998ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
38562026-03-10T20:08:09.999ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database result = Ok(())
38572026-03-10T20:08:10.002ZINFOtest_retry_backoff (BackgroundTasks): webhook dispatching completed successfully background_task = alert_dispatcher events_dispatched = 1 events_without_receivers = 0
38582026-03-10T20:08:10.002ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 183.395574ms iteration = 2 status = {"dispatched":[{"alert_id":"5b38ce09-0fe3-4310-a5b1-e7643a39c1de","dispatched":1,"subscribed":1}],"errors":[],"glob_version":"237.0.0","globs_reprocessed":{},"no_receivers":[]}
38592026-03-10T20:08:10.002ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = webhook_deliverator iteration = 2 reason = Signaled
38602026-03-10T20:08:10.002ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database
38612026-03-10T20:08:10.002ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
38622026-03-10T20:08:10.002ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(())
38632026-03-10T20:08:10.008ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
38642026-03-10T20:08:10.008ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
38652026-03-10T20:08:10.008ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
38662026-03-10T20:08:10.018ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/alert_dispatcher
38672026-03-10T20:08:10.018ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 17215941-658e-4653-8ff0-ee41c2b83135 uri = /bgtasks/view/alert_dispatcher
38682026-03-10T20:08:10.018ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 17215941-658e-4653-8ff0-ee41c2b83135 resource = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38692026-03-10T20:08:10.019ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 17215941-658e-4653-8ff0-ee41c2b83135 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38702026-03-10T20:08:10.019ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 17215941-658e-4653-8ff0-ee41c2b83135 resource = Database type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38712026-03-10T20:08:10.019ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 17215941-658e-4653-8ff0-ee41c2b83135 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38722026-03-10T20:08:10.019ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 17215941-658e-4653-8ff0-ee41c2b83135 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38732026-03-10T20:08:10.021ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 17215941-658e-4653-8ff0-ee41c2b83135 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38742026-03-10T20:08:10.024ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 17215941-658e-4653-8ff0-ee41c2b83135 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38752026-03-10T20:08:10.024ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5872 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 17215941-658e-4653-8ff0-ee41c2b83135 response_code = 200 uri = /bgtasks/view/alert_dispatcher
38762026-03-10T20:08:10.024ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
38772026-03-10T20:08:10.025ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:50070/bgtasks/activate
38782026-03-10T20:08:10.025ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 547ea0bd-4506-421c-9222-a82890ba722b uri = /bgtasks/activate
38792026-03-10T20:08:10.025ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 547ea0bd-4506-421c-9222-a82890ba722b resource = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
38802026-03-10T20:08:10.025ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 547ea0bd-4506-421c-9222-a82890ba722b resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
38812026-03-10T20:08:10.025ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 547ea0bd-4506-421c-9222-a82890ba722b resource = Database type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
38822026-03-10T20:08:10.025ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 547ea0bd-4506-421c-9222-a82890ba722b roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
38832026-03-10T20:08:10.025ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 547ea0bd-4506-421c-9222-a82890ba722b resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
38842026-03-10T20:08:10.027ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 547ea0bd-4506-421c-9222-a82890ba722b roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
38852026-03-10T20:08:10.029ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 547ea0bd-4506-421c-9222-a82890ba722b resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
38862026-03-10T20:08:10.029ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = alert_dispatcher iteration = 3 reason = Signaled
38872026-03-10T20:08:10.029ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database
38882026-03-10T20:08:10.029ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
38892026-03-10T20:08:10.029ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database result = Ok(())
38902026-03-10T20:08:10.030ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5263 local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 547ea0bd-4506-421c-9222-a82890ba722b response_code = 204 uri = /bgtasks/activate
38912026-03-10T20:08:10.030ZTRACtest_retry_backoff (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38922026-03-10T20:08:10.030ZTRACtest_retry_backoff (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38932026-03-10T20:08:10.030ZINFOtest_retry_backoff (lockstep client test context): client received response status = 204
38942026-03-10T20:08:10.030ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/alert_dispatcher
38952026-03-10T20:08:10.030ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e374265d-ae6a-499e-87e0-55fa1175d95b uri = /bgtasks/view/alert_dispatcher
38962026-03-10T20:08:10.031ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e374265d-ae6a-499e-87e0-55fa1175d95b resource = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38972026-03-10T20:08:10.031ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e374265d-ae6a-499e-87e0-55fa1175d95b resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38982026-03-10T20:08:10.031ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e374265d-ae6a-499e-87e0-55fa1175d95b resource = Database type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
38992026-03-10T20:08:10.031ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e374265d-ae6a-499e-87e0-55fa1175d95b roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39002026-03-10T20:08:10.031ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e374265d-ae6a-499e-87e0-55fa1175d95b resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39012026-03-10T20:08:10.032ZDEBGtest_retry_backoff (producer-registration-task): will register with Nexus at 127.0.0.1:63060
39022026-03-10T20:08:10.071ZDEBGtest_retry_backoff (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:63060/metrics/producers
39032026-03-10T20:08:10.071ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e374265d-ae6a-499e-87e0-55fa1175d95b roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39042026-03-10T20:08:10.075ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e374265d-ae6a-499e-87e0-55fa1175d95b resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39052026-03-10T20:08:10.075ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 44714 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e374265d-ae6a-499e-87e0-55fa1175d95b response_code = 200 uri = /bgtasks/view/alert_dispatcher
39062026-03-10T20:08:10.076ZINFOtest_retry_backoff (dropshot_internal): accepted connection local_addr = 127.0.0.1:63060 remote_addr = 127.0.0.1:57596
39072026-03-10T20:08:10.076ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
39082026-03-10T20:08:10.077ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:57596 req_id = 5534efb4-fd2b-4733-88c4-ff0774cac9cb uri = /metrics/producers
39092026-03-10T20:08:10.077ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:57596 req_id = 5534efb4-fd2b-4733-88c4-ff0774cac9cb uri = /metrics/producers version = 1.0.0
39102026-03-10T20:08:10.077ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:57596 req_id = 5534efb4-fd2b-4733-88c4-ff0774cac9cb resource = Database type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
39112026-03-10T20:08:10.077ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:57596 req_id = 5534efb4-fd2b-4733-88c4-ff0774cac9cb roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
39122026-03-10T20:08:10.077ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:57596 req_id = 5534efb4-fd2b-4733-88c4-ff0774cac9cb resource = Database result = Ok(()) type = user_builtin uri = /metrics/producers user_builtin_id = 001de000-05e4-4000-8000-000000000002
39132026-03-10T20:08:10.078ZTRACtest_retry_backoff (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = alert_dispatcher
39142026-03-10T20:08:10.078ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database
39152026-03-10T20:08:10.078ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
39162026-03-10T20:08:10.079ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = alert_dispatcher resource = Database result = Ok(())
39172026-03-10T20:08:10.081ZTRACtest_retry_backoff (BackgroundTasks): webhook dispatching completed successfully background_task = alert_dispatcher events_dispatched = 0 events_without_receivers = 0
39182026-03-10T20:08:10.081ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 51.569155ms iteration = 3 status = {"dispatched":[],"errors":[],"glob_version":"237.0.0","globs_reprocessed":{},"no_receivers":[]}
39192026-03-10T20:08:10.082ZINFOtest_retry_backoff (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = 913233fe-92a8-4635-9572-183f495429c4
39202026-03-10T20:08:10.082ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 5561 local_addr = 127.0.0.1:63060 method = POST remote_addr = 127.0.0.1:57596 req_id = 5534efb4-fd2b-4733-88c4-ff0774cac9cb response_code = 201 uri = /metrics/producers
39212026-03-10T20:08:10.082ZDEBGtest_retry_backoff (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "5534efb4-fd2b-4733-88c4-ff0774cac9cb", "content-length": "41", "date": "Tue, 10 Mar 2026 20:08:10 GMT"} })
39222026-03-10T20:08:10.083ZDEBGtest_retry_backoff (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
39232026-03-10T20:08:10.084ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39242026-03-10T20:08:10.084ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39252026-03-10T20:08:10.084ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39262026-03-10T20:08:10.126ZTRACtest_retry_backoff (BackgroundTasks): webhook event delivery attempt started alert_class = test.foo alert_id = 5b38ce09-0fe3-4310-a5b1-e7643a39c1de attempt = 1 background_task = webhook_deliverator delivery_id = 13901bf0-04f6-4b8c-a2c1-278327927fd7 rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39272026-03-10T20:08:10.127ZWARNtest_retry_backoff (BackgroundTasks): webhook receiver endpoint returned an HTTP error alert_class = test.foo alert_id = 5b38ce09-0fe3-4310-a5b1-e7643a39c1de background_task = webhook_deliverator delivery_id = 13901bf0-04f6-4b8c-a2c1-278327927fd7 delivery_trigger = alert response_duration = 1.334614ms response_status = 500 rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39282026-03-10T20:08:10.127ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39292026-03-10T20:08:10.127ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39302026-03-10T20:08:10.128ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39312026-03-10T20:08:10.128ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/alert_dispatcher
39322026-03-10T20:08:10.129ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 613bb3e9-da20-44a9-84f5-402325c5f521 uri = /bgtasks/view/alert_dispatcher
39332026-03-10T20:08:10.129ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 613bb3e9-da20-44a9-84f5-402325c5f521 resource = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39342026-03-10T20:08:10.129ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 613bb3e9-da20-44a9-84f5-402325c5f521 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39352026-03-10T20:08:10.129ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 613bb3e9-da20-44a9-84f5-402325c5f521 resource = Database type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39362026-03-10T20:08:10.129ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 613bb3e9-da20-44a9-84f5-402325c5f521 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39372026-03-10T20:08:10.129ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 613bb3e9-da20-44a9-84f5-402325c5f521 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39382026-03-10T20:08:10.131ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 613bb3e9-da20-44a9-84f5-402325c5f521 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39392026-03-10T20:08:10.134ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 613bb3e9-da20-44a9-84f5-402325c5f521 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/alert_dispatcher user_builtin_id = 001de000-05e4-4000-8000-000000000002
39402026-03-10T20:08:10.134ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5551 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 613bb3e9-da20-44a9-84f5-402325c5f521 response_code = 200 uri = /bgtasks/view/alert_dispatcher
39412026-03-10T20:08:10.134ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
39422026-03-10T20:08:10.135ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
39432026-03-10T20:08:10.135ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f63beb8f-b98b-4799-9335-05bfe14aeaa2 uri = /bgtasks/view/webhook_deliverator
39442026-03-10T20:08:10.135ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f63beb8f-b98b-4799-9335-05bfe14aeaa2 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39452026-03-10T20:08:10.135ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f63beb8f-b98b-4799-9335-05bfe14aeaa2 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39462026-03-10T20:08:10.135ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f63beb8f-b98b-4799-9335-05bfe14aeaa2 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39472026-03-10T20:08:10.135ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f63beb8f-b98b-4799-9335-05bfe14aeaa2 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39482026-03-10T20:08:10.135ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f63beb8f-b98b-4799-9335-05bfe14aeaa2 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39492026-03-10T20:08:10.137ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f63beb8f-b98b-4799-9335-05bfe14aeaa2 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39502026-03-10T20:08:10.140ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f63beb8f-b98b-4799-9335-05bfe14aeaa2 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39512026-03-10T20:08:10.140ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5428 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f63beb8f-b98b-4799-9335-05bfe14aeaa2 response_code = 200 uri = /bgtasks/view/webhook_deliverator
39522026-03-10T20:08:10.140ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
39532026-03-10T20:08:10.141ZINFOtest_retry_backoff (lockstep client test context): waiting for webhook_deliverator to go idle
39542026-03-10T20:08:10.173ZINFOtest_retry_backoff (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator num_receivers = 1
39552026-03-10T20:08:10.173ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 171.632362ms iteration = 2 status = {"by_rx":{"2b6fbbc6-d38e-4d46-a5cd-293e75593af0":{"already_delivered":0,"delivered_ok":0,"delivery_errors":{},"error":null,"failed_deliveries":[{"alert_id":"5b38ce09-0fe3-4310-a5b1-e7643a39c1de","attempt":1,"delivery_id":"13901bf0-04f6-4b8c-a2c1-278327927fd7","response_duration":[0,1334614],"response_status":500,"result":"failed_http_error"}],"in_progress":0,"ready":1}},"error":null}
39562026-03-10T20:08:10.192ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
39572026-03-10T20:08:10.192ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6db64905-2655-4192-b53d-5fbb98fb883b uri = /bgtasks/view/webhook_deliverator
39582026-03-10T20:08:10.192ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6db64905-2655-4192-b53d-5fbb98fb883b resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39592026-03-10T20:08:10.192ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6db64905-2655-4192-b53d-5fbb98fb883b resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39602026-03-10T20:08:10.193ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6db64905-2655-4192-b53d-5fbb98fb883b resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39612026-03-10T20:08:10.193ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6db64905-2655-4192-b53d-5fbb98fb883b roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39622026-03-10T20:08:10.193ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6db64905-2655-4192-b53d-5fbb98fb883b resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39632026-03-10T20:08:10.195ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6db64905-2655-4192-b53d-5fbb98fb883b roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39642026-03-10T20:08:10.198ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6db64905-2655-4192-b53d-5fbb98fb883b resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39652026-03-10T20:08:10.199ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 6237 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6db64905-2655-4192-b53d-5fbb98fb883b response_code = 200 uri = /bgtasks/view/webhook_deliverator
39662026-03-10T20:08:10.199ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
39672026-03-10T20:08:10.199ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:50070/bgtasks/activate
39682026-03-10T20:08:10.199ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 6415a86a-b98a-4e06-b69b-ac854fb14b9e uri = /bgtasks/activate
39692026-03-10T20:08:10.199ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 6415a86a-b98a-4e06-b69b-ac854fb14b9e resource = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
39702026-03-10T20:08:10.199ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 6415a86a-b98a-4e06-b69b-ac854fb14b9e resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
39712026-03-10T20:08:10.199ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 6415a86a-b98a-4e06-b69b-ac854fb14b9e resource = Database type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
39722026-03-10T20:08:10.199ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 6415a86a-b98a-4e06-b69b-ac854fb14b9e roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
39732026-03-10T20:08:10.200ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 6415a86a-b98a-4e06-b69b-ac854fb14b9e resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
39742026-03-10T20:08:10.201ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 6415a86a-b98a-4e06-b69b-ac854fb14b9e roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
39752026-03-10T20:08:10.203ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 6415a86a-b98a-4e06-b69b-ac854fb14b9e resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
39762026-03-10T20:08:10.203ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = webhook_deliverator iteration = 3 reason = Signaled
39772026-03-10T20:08:10.203ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database
39782026-03-10T20:08:10.203ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
39792026-03-10T20:08:10.204ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(())
39802026-03-10T20:08:10.204ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5049 local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 6415a86a-b98a-4e06-b69b-ac854fb14b9e response_code = 204 uri = /bgtasks/activate
39812026-03-10T20:08:10.204ZINFOtest_retry_backoff (lockstep client test context): client received response status = 204
39822026-03-10T20:08:10.204ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
39832026-03-10T20:08:10.205ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e6ca57bd-1f72-4a3b-bd0c-48c972473018 uri = /bgtasks/view/webhook_deliverator
39842026-03-10T20:08:10.205ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e6ca57bd-1f72-4a3b-bd0c-48c972473018 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39852026-03-10T20:08:10.205ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e6ca57bd-1f72-4a3b-bd0c-48c972473018 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39862026-03-10T20:08:10.205ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e6ca57bd-1f72-4a3b-bd0c-48c972473018 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39872026-03-10T20:08:10.205ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e6ca57bd-1f72-4a3b-bd0c-48c972473018 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39882026-03-10T20:08:10.205ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e6ca57bd-1f72-4a3b-bd0c-48c972473018 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39892026-03-10T20:08:10.207ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e6ca57bd-1f72-4a3b-bd0c-48c972473018 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39902026-03-10T20:08:10.210ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e6ca57bd-1f72-4a3b-bd0c-48c972473018 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
39912026-03-10T20:08:10.210ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5547 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e6ca57bd-1f72-4a3b-bd0c-48c972473018 response_code = 200 uri = /bgtasks/view/webhook_deliverator
39922026-03-10T20:08:10.210ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
39932026-03-10T20:08:10.213ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39942026-03-10T20:08:10.213ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39952026-03-10T20:08:10.213ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
39962026-03-10T20:08:10.253ZINFOtest_retry_backoff (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator num_receivers = 1
39972026-03-10T20:08:10.253ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 50.14962ms iteration = 3 status = {"by_rx":{"2b6fbbc6-d38e-4d46-a5cd-293e75593af0":{"already_delivered":0,"delivered_ok":0,"delivery_errors":{},"error":null,"failed_deliveries":[],"in_progress":0,"ready":0}},"error":null}
39982026-03-10T20:08:10.262ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
39992026-03-10T20:08:10.262ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dc403bfe-8393-4f51-8d91-cf688222f935 uri = /bgtasks/view/webhook_deliverator
40002026-03-10T20:08:10.262ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dc403bfe-8393-4f51-8d91-cf688222f935 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
40012026-03-10T20:08:10.262ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dc403bfe-8393-4f51-8d91-cf688222f935 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
40022026-03-10T20:08:10.262ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dc403bfe-8393-4f51-8d91-cf688222f935 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
40032026-03-10T20:08:10.263ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dc403bfe-8393-4f51-8d91-cf688222f935 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
40042026-03-10T20:08:10.263ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dc403bfe-8393-4f51-8d91-cf688222f935 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
40052026-03-10T20:08:10.265ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dc403bfe-8393-4f51-8d91-cf688222f935 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
40062026-03-10T20:08:10.268ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dc403bfe-8393-4f51-8d91-cf688222f935 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
40072026-03-10T20:08:10.268ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5934 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dc403bfe-8393-4f51-8d91-cf688222f935 response_code = 200 uri = /bgtasks/view/webhook_deliverator
40082026-03-10T20:08:10.268ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
40092026-03-10T20:08:10.269ZINFOtest_retry_backoff (external client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:46842/v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40102026-03-10T20:08:10.269ZTRACtest_retry_backoff (dropshot_external): incoming request local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40112026-03-10T20:08:10.269ZDEBGtest_retry_backoff (dropshot_external): determined request API version local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10 version = 2026030200.0.0
40122026-03-10T20:08:10.269ZTRACtest_retry_backoff (dropshot_external): authn: trying Spoof local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40132026-03-10T20:08:10.269ZDEBGtest_retry_backoff (dropshot_external): looking up silo for user local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad silo_user_id = 001de000-05e4-4000-8000-000000004007 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40142026-03-10T20:08:10.269ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40152026-03-10T20:08:10.269ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40162026-03-10T20:08:10.270ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40172026-03-10T20:08:10.272ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40182026-03-10T20:08:10.272ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40192026-03-10T20:08:10.272ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40202026-03-10T20:08:10.274ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007 (silo_user), lookup_type: ById(001de000-05e4-4000-8000-000000004007) }
40212026-03-10T20:08:10.274ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
40222026-03-10T20:08:10.274ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40232026-03-10T20:08:10.274ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40242026-03-10T20:08:10.274ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40252026-03-10T20:08:10.276ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40262026-03-10T20:08:10.276ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40272026-03-10T20:08:10.276ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40282026-03-10T20:08:10.276ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40292026-03-10T20:08:10.277ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
40302026-03-10T20:08:10.279ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007 (silo_user), lookup_type: ById(001de000-05e4-4000-8000-000000004007) } result = Ok(())
40312026-03-10T20:08:10.279ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40322026-03-10T20:08:10.280ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40332026-03-10T20:08:10.280ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40342026-03-10T20:08:10.281ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }
40352026-03-10T20:08:10.281ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
40362026-03-10T20:08:10.281ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40372026-03-10T20:08:10.281ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40382026-03-10T20:08:10.281ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40392026-03-10T20:08:10.283ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40402026-03-10T20:08:10.283ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40412026-03-10T20:08:10.283ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40422026-03-10T20:08:10.283ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40432026-03-10T20:08:10.284ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
40442026-03-10T20:08:10.286ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) } result = Ok(())
40452026-03-10T20:08:10.286ZTRACtest_retry_backoff (dropshot_external): authn result: Ok(Context { kind: Authenticated(Details { actor: Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }, device_token_expiration: None, credential_id: None }, Some(SiloAuthnPolicy { mapped_fleet_roles: {} })), schemes_tried: [Spoof] }) local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40462026-03-10T20:08:10.286ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40472026-03-10T20:08:10.286ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40482026-03-10T20:08:10.287ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40492026-03-10T20:08:10.288ZTRACtest_retry_backoff (dropshot_external): authorize begin action = ListChildren actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad resource = AlertReceiver { parent: Fleet, key: 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver), lookup_type: ById(2b6fbbc6-d38e-4d46-a5cd-293e75593af0) } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40502026-03-10T20:08:10.288ZTRACtest_retry_backoff (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. } authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40512026-03-10T20:08:10.288ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40522026-03-10T20:08:10.289ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40532026-03-10T20:08:10.289ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40542026-03-10T20:08:10.291ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40552026-03-10T20:08:10.295ZDEBGtest_retry_backoff (dropshot_external): authorize result action = ListChildren actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad resource = AlertReceiver { parent: Fleet, key: 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver), lookup_type: ById(2b6fbbc6-d38e-4d46-a5cd-293e75593af0) } result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40562026-03-10T20:08:10.295ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40572026-03-10T20:08:10.295ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40582026-03-10T20:08:10.295ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40592026-03-10T20:08:10.301ZINFOtest_retry_backoff (dropshot_external): request completed latency_us = 31975 local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = b7fc6903-cb56-4306-8565-579a3c39e0ad response_code = 200 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
40602026-03-10T20:08:10.301ZINFOtest_retry_backoff (external client test context): client received response status = 200
40612026-03-10T20:08:10.302ZINFOtest_retry_backoff (external client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:46842/v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
40622026-03-10T20:08:10.302ZTRACtest_retry_backoff (dropshot_external): incoming request local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
40632026-03-10T20:08:10.302ZDEBGtest_retry_backoff (dropshot_external): determined request API version local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ== version = 2026030200.0.0
40642026-03-10T20:08:10.302ZTRACtest_retry_backoff (dropshot_external): authn: trying Spoof local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
40652026-03-10T20:08:10.302ZDEBGtest_retry_backoff (dropshot_external): looking up silo for user local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 silo_user_id = 001de000-05e4-4000-8000-000000004007 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
40662026-03-10T20:08:10.302ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40672026-03-10T20:08:10.302ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40682026-03-10T20:08:10.303ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40692026-03-10T20:08:10.304ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40702026-03-10T20:08:10.304ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40712026-03-10T20:08:10.305ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40722026-03-10T20:08:10.306ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007 (silo_user), lookup_type: ById(001de000-05e4-4000-8000-000000004007) }
40732026-03-10T20:08:10.307ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
40742026-03-10T20:08:10.307ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40752026-03-10T20:08:10.307ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40762026-03-10T20:08:10.307ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40772026-03-10T20:08:10.309ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40782026-03-10T20:08:10.310ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40792026-03-10T20:08:10.310ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40802026-03-10T20:08:10.310ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40812026-03-10T20:08:10.312ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
40822026-03-10T20:08:10.314ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007 (silo_user), lookup_type: ById(001de000-05e4-4000-8000-000000004007) } result = Ok(())
40832026-03-10T20:08:10.314ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40842026-03-10T20:08:10.314ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40852026-03-10T20:08:10.314ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40862026-03-10T20:08:10.316ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }
40872026-03-10T20:08:10.316ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
40882026-03-10T20:08:10.316ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40892026-03-10T20:08:10.316ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40902026-03-10T20:08:10.316ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40912026-03-10T20:08:10.318ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40922026-03-10T20:08:10.318ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
40932026-03-10T20:08:10.318ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
40942026-03-10T20:08:10.318ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
40952026-03-10T20:08:10.319ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
40962026-03-10T20:08:10.322ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) } result = Ok(())
40972026-03-10T20:08:10.322ZTRACtest_retry_backoff (dropshot_external): authn result: Ok(Context { kind: Authenticated(Details { actor: Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }, device_token_expiration: None, credential_id: None }, Some(SiloAuthnPolicy { mapped_fleet_roles: {} })), schemes_tried: [Spoof] }) local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
40982026-03-10T20:08:10.322ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
40992026-03-10T20:08:10.322ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41002026-03-10T20:08:10.322ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41012026-03-10T20:08:10.324ZTRACtest_retry_backoff (dropshot_external): authorize begin action = ListChildren actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 resource = AlertReceiver { parent: Fleet, key: 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver), lookup_type: ById(2b6fbbc6-d38e-4d46-a5cd-293e75593af0) } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41022026-03-10T20:08:10.324ZTRACtest_retry_backoff (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. } authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41032026-03-10T20:08:10.324ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41042026-03-10T20:08:10.324ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41052026-03-10T20:08:10.324ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41062026-03-10T20:08:10.326ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41072026-03-10T20:08:10.331ZDEBGtest_retry_backoff (dropshot_external): authorize result action = ListChildren actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 resource = AlertReceiver { parent: Fleet, key: 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver), lookup_type: ById(2b6fbbc6-d38e-4d46-a5cd-293e75593af0) } result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41082026-03-10T20:08:10.331ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41092026-03-10T20:08:10.331ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41102026-03-10T20:08:10.331ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41112026-03-10T20:08:10.335ZINFOtest_retry_backoff (dropshot_external): request completed latency_us = 33064 local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 0311047a-a74e-4674-ad80-a38339892961 response_code = 200 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
41122026-03-10T20:08:10.335ZINFOtest_retry_backoff (external client test context): client received response status = 200
41132026-03-10T20:08:10.336ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
41142026-03-10T20:08:10.336ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6c694f2b-de82-492c-a059-e8908ecc816f uri = /bgtasks/view/webhook_deliverator
41152026-03-10T20:08:10.336ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6c694f2b-de82-492c-a059-e8908ecc816f resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41162026-03-10T20:08:10.336ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6c694f2b-de82-492c-a059-e8908ecc816f resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41172026-03-10T20:08:10.336ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6c694f2b-de82-492c-a059-e8908ecc816f resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41182026-03-10T20:08:10.336ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6c694f2b-de82-492c-a059-e8908ecc816f roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41192026-03-10T20:08:10.336ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6c694f2b-de82-492c-a059-e8908ecc816f resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41202026-03-10T20:08:10.338ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6c694f2b-de82-492c-a059-e8908ecc816f roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41212026-03-10T20:08:10.345ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6c694f2b-de82-492c-a059-e8908ecc816f resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41222026-03-10T20:08:10.345ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 9460 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 6c694f2b-de82-492c-a059-e8908ecc816f response_code = 200 uri = /bgtasks/view/webhook_deliverator
41232026-03-10T20:08:10.345ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
41242026-03-10T20:08:10.346ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:50070/bgtasks/activate
41252026-03-10T20:08:10.346ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 42193ffa-eab7-4b50-be3d-b8c0c472a060 uri = /bgtasks/activate
41262026-03-10T20:08:10.346ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 42193ffa-eab7-4b50-be3d-b8c0c472a060 resource = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
41272026-03-10T20:08:10.346ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 42193ffa-eab7-4b50-be3d-b8c0c472a060 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
41282026-03-10T20:08:10.346ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 42193ffa-eab7-4b50-be3d-b8c0c472a060 resource = Database type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
41292026-03-10T20:08:10.346ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 42193ffa-eab7-4b50-be3d-b8c0c472a060 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
41302026-03-10T20:08:10.346ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 42193ffa-eab7-4b50-be3d-b8c0c472a060 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
41312026-03-10T20:08:10.348ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 42193ffa-eab7-4b50-be3d-b8c0c472a060 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
41322026-03-10T20:08:10.350ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 42193ffa-eab7-4b50-be3d-b8c0c472a060 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
41332026-03-10T20:08:10.350ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = webhook_deliverator iteration = 4 reason = Signaled
41342026-03-10T20:08:10.350ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database
41352026-03-10T20:08:10.350ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
41362026-03-10T20:08:10.351ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(())
41372026-03-10T20:08:10.351ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5426 local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 42193ffa-eab7-4b50-be3d-b8c0c472a060 response_code = 204 uri = /bgtasks/activate
41382026-03-10T20:08:10.351ZINFOtest_retry_backoff (lockstep client test context): client received response status = 204
41392026-03-10T20:08:10.351ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
41402026-03-10T20:08:10.351ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f71ac640-4366-4884-8bb4-a9a6cf2208cf uri = /bgtasks/view/webhook_deliverator
41412026-03-10T20:08:10.352ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f71ac640-4366-4884-8bb4-a9a6cf2208cf resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41422026-03-10T20:08:10.352ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f71ac640-4366-4884-8bb4-a9a6cf2208cf resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41432026-03-10T20:08:10.352ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f71ac640-4366-4884-8bb4-a9a6cf2208cf resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41442026-03-10T20:08:10.352ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f71ac640-4366-4884-8bb4-a9a6cf2208cf roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41452026-03-10T20:08:10.352ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f71ac640-4366-4884-8bb4-a9a6cf2208cf resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41462026-03-10T20:08:10.354ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f71ac640-4366-4884-8bb4-a9a6cf2208cf roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41472026-03-10T20:08:10.357ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f71ac640-4366-4884-8bb4-a9a6cf2208cf resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41482026-03-10T20:08:10.357ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5377 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = f71ac640-4366-4884-8bb4-a9a6cf2208cf response_code = 200 uri = /bgtasks/view/webhook_deliverator
41492026-03-10T20:08:10.357ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
41502026-03-10T20:08:10.358ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
41512026-03-10T20:08:10.358ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
41522026-03-10T20:08:10.358ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
41532026-03-10T20:08:10.361ZINFOtest_retry_backoff (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator num_receivers = 1
41542026-03-10T20:08:10.361ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 10.536837ms iteration = 4 status = {"by_rx":{"2b6fbbc6-d38e-4d46-a5cd-293e75593af0":{"already_delivered":0,"delivered_ok":0,"delivery_errors":{},"error":null,"failed_deliveries":[],"in_progress":0,"ready":0}},"error":null}
41552026-03-10T20:08:10.408ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
41562026-03-10T20:08:10.409ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dbb886d3-0ab4-4bb5-afcc-b04f9f00cb74 uri = /bgtasks/view/webhook_deliverator
41572026-03-10T20:08:10.409ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dbb886d3-0ab4-4bb5-afcc-b04f9f00cb74 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41582026-03-10T20:08:10.409ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dbb886d3-0ab4-4bb5-afcc-b04f9f00cb74 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41592026-03-10T20:08:10.409ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dbb886d3-0ab4-4bb5-afcc-b04f9f00cb74 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41602026-03-10T20:08:10.409ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dbb886d3-0ab4-4bb5-afcc-b04f9f00cb74 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41612026-03-10T20:08:10.409ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dbb886d3-0ab4-4bb5-afcc-b04f9f00cb74 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41622026-03-10T20:08:10.411ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dbb886d3-0ab4-4bb5-afcc-b04f9f00cb74 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41632026-03-10T20:08:10.414ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dbb886d3-0ab4-4bb5-afcc-b04f9f00cb74 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
41642026-03-10T20:08:10.414ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5700 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = dbb886d3-0ab4-4bb5-afcc-b04f9f00cb74 response_code = 200 uri = /bgtasks/view/webhook_deliverator
41652026-03-10T20:08:10.414ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
41662026-03-10T20:08:10.562ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = reconfigurator_config_watcher iteration = 2 reason = Timeout
41672026-03-10T20:08:10.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig
41682026-03-10T20:08:10.562ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = reconfigurator_config_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41692026-03-10T20:08:10.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
41702026-03-10T20:08:10.562ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
41712026-03-10T20:08:10.563ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
41722026-03-10T20:08:10.565ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41732026-03-10T20:08:10.569ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig result = Ok(())
41742026-03-10T20:08:10.569ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
41752026-03-10T20:08:10.569ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
41762026-03-10T20:08:10.569ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
41772026-03-10T20:08:10.571ZDEBGtest_retry_backoff (BackgroundTasks): reconfigurator config load complete background_task = reconfigurator_config_watcher switches = Loaded(ReconfiguratorConfigView { version: 1, config: ReconfiguratorConfig { planner_enabled: false, planner_config: PlannerConfig { add_zones_with_mupdate_override: false }, tuf_repo_pruner_enabled: true }, time_modified: 2026-03-10T20:08:05.458980Z })
41782026-03-10T20:08:10.571ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = reconfigurator_config_watcher elapsed = 8.938975ms iteration = 2 status = {"config_updated":false}
41792026-03-10T20:08:11.029ZTRACtest_retry_backoff (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41802026-03-10T20:08:11.029ZTRACtest_retry_backoff (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41812026-03-10T20:08:11.443ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
41822026-03-10T20:08:11.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
41832026-03-10T20:08:11.443ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9b3dede8-3dc7-4ba0-9643-46d2692be599 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
41842026-03-10T20:08:11.443ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9b3dede8-3dc7-4ba0-9643-46d2692be599 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
41852026-03-10T20:08:11.444ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9b3dede8-3dc7-4ba0-9643-46d2692be599 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
41862026-03-10T20:08:11.444ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9b3dede8-3dc7-4ba0-9643-46d2692be599 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
41872026-03-10T20:08:11.444ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9b3dede8-3dc7-4ba0-9643-46d2692be599 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
41882026-03-10T20:08:11.447ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3203 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9b3dede8-3dc7-4ba0-9643-46d2692be599 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
41892026-03-10T20:08:11.447ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b3dede8-3dc7-4ba0-9643-46d2692be599", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:11 GMT"} })
41902026-03-10T20:08:11.447ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
41912026-03-10T20:08:11.447ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9e2f8f0e-2842-4346-bfc3-79ecb004058e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
41922026-03-10T20:08:11.447ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9e2f8f0e-2842-4346-bfc3-79ecb004058e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
41932026-03-10T20:08:11.447ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9e2f8f0e-2842-4346-bfc3-79ecb004058e resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
41942026-03-10T20:08:11.447ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9e2f8f0e-2842-4346-bfc3-79ecb004058e roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
41952026-03-10T20:08:11.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9e2f8f0e-2842-4346-bfc3-79ecb004058e resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
41962026-03-10T20:08:11.450ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2858 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9e2f8f0e-2842-4346-bfc3-79ecb004058e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
41972026-03-10T20:08:11.450ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e2f8f0e-2842-4346-bfc3-79ecb004058e", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:11 GMT"} })
41982026-03-10T20:08:11.450ZDEBGtest_retry_backoff (oximeter-agent): registered new metric producer address = 127.0.0.1:33574 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
41992026-03-10T20:08:11.450ZDEBGtest_retry_backoff (oximeter-agent): registered new metric producer address = [::1]:41614 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
42002026-03-10T20:08:11.450ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
42012026-03-10T20:08:11.486ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
42022026-03-10T20:08:11.522ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42032026-03-10T20:08:11.522ZDEBGtest_retry_backoff (collection-task): reporting oximeter self-collection statistics collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
42042026-03-10T20:08:11.523ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
42052026-03-10T20:08:11.523ZDEBGtest_retry_backoff (collection-task): reporting oximeter self-collection statistics collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42062026-03-10T20:08:11.523ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42072026-03-10T20:08:11.523ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
42082026-03-10T20:08:11.524ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42092026-03-10T20:08:11.524ZINFOtest_retry_backoff (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:1025 local_addr = 127.0.0.1:33574 remote_addr = 127.0.0.1:51522
42102026-03-10T20:08:11.524ZTRACtest_retry_backoff (dropshot): incoming request local_addr = 127.0.0.1:33574 method = GET remote_addr = 127.0.0.1:51522 req_id = 45084fcb-1d58-404a-94c4-1cbe9c715011 uri = /913233fe-92a8-4635-9572-183f495429c4
42112026-03-10T20:08:11.530ZINFOtest_retry_backoff (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867 latency_us = 6147 local_addr = 127.0.0.1:33574 method = GET remote_addr = 127.0.0.1:51522 req_id = 45084fcb-1d58-404a-94c4-1cbe9c715011 response_code = 200 uri = /913233fe-92a8-4635-9572-183f495429c4
42122026-03-10T20:08:11.531ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
42132026-03-10T20:08:11.531ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42142026-03-10T20:08:11.531ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42152026-03-10T20:08:11.531ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42162026-03-10T20:08:11.531ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42172026-03-10T20:08:11.542ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
42182026-03-10T20:08:11.542ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
42192026-03-10T20:08:11.542ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
42202026-03-10T20:08:11.542ZDEBGtest_retry_backoff (results-sink): inserting 25 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42212026-03-10T20:08:11.542ZDEBGtest_retry_backoff (results-sink-cluster): inserting 25 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42222026-03-10T20:08:11.542ZDEBGtest_retry_backoff (clickhouse-client): unrolling 25 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42232026-03-10T20:08:11.543ZDEBGtest_retry_backoff (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42242026-03-10T20:08:11.543ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
42252026-03-10T20:08:11.544ZTRACtest_retry_backoff (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42262026-03-10T20:08:11.544ZDEBGtest_retry_backoff (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f schema = TimeseriesSchema { timeseries_name: TimeseriesName("oximeter_collector:collections"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "base_route", field_type: String, source: Metric, description: "" }, FieldSchema { name: "collector_id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "collector_ip", field_type: IpAddr, source: Target, description: "" }, FieldSchema { name: "collector_port", field_type: U16, source: Target, description: "" }, FieldSchema { name: "producer_id", field_type: Uuid, source: Metric, description: "" }, FieldSchema { name: "producer_ip", field_type: IpAddr, source: Metric, description: "" }, FieldSchema { name: "producer_port", field_type: U16, source: Metric, description: "" }}, datum_type: CumulativeU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-03-10T20:08:11.542959865Z } timeseries_name = oximeter_collector:collections
42272026-03-10T20:08:11.545ZDEBGtest_retry_backoff (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42282026-03-10T20:08:11.545ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('oximeter_collector:collections') FORMAT Native;
42292026-03-10T20:08:11.547ZTRACtest_retry_backoff (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42302026-03-10T20:08:11.547ZDEBGtest_retry_backoff (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2026-03-10T20:08:11.545802548Z } timeseries_name = integration_target:integration_metric
42312026-03-10T20:08:11.547ZDEBGtest_retry_backoff (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42322026-03-10T20:08:11.547ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric', 'oximeter_collector:collections') FORMAT Native;
42332026-03-10T20:08:11.548ZTRACtest_retry_backoff (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42342026-03-10T20:08:11.548ZDEBGtest_retry_backoff (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2026-03-10T20:08:11.547513309Z } timeseries_name = http_service:request_latency_histogram
42352026-03-10T20:08:11.553ZDEBGtest_retry_backoff (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42362026-03-10T20:08:11.553ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric', 'oximeter_collector:collections') FORMAT Native;
42372026-03-10T20:08:11.557ZTRACtest_retry_backoff (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42382026-03-10T20:08:11.557ZDEBGtest_retry_backoff (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2026-03-10T20:08:11.553780155Z } timeseries_name = collection_target:cpus_provisioned
42392026-03-10T20:08:11.558ZDEBGtest_retry_backoff (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42402026-03-10T20:08:11.558ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric', 'oximeter_collector:collections') FORMAT Native;
42412026-03-10T20:08:11.558ZTRACtest_retry_backoff (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42422026-03-10T20:08:11.558ZDEBGtest_retry_backoff (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2026-03-10T20:08:11.555744111Z } timeseries_name = collection_target:ram_provisioned
42432026-03-10T20:08:11.558ZDEBGtest_retry_backoff (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42442026-03-10T20:08:11.558ZTRACtest_retry_backoff (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric', 'oximeter_collector:collections') FORMAT Native;
42452026-03-10T20:08:11.558ZTRACtest_retry_backoff (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42462026-03-10T20:08:11.558ZDEBGtest_retry_backoff (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2026-03-10T20:08:11.557726072Z } timeseries_name = collection_target:virtual_disk_space_provisioned
42472026-03-10T20:08:11.563ZDEBGtest_retry_backoff (clickhouse-client): inserting 6 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42482026-03-10T20:08:11.563ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 6 n_rows = 6 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
42492026-03-10T20:08:11.566ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 4 sql = INSERT INTO oximeter.fields_ipaddr FORMAT Native
42502026-03-10T20:08:11.567ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 4 table_name = fields_ipaddr
42512026-03-10T20:08:11.567ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 24 sql = INSERT INTO oximeter.fields_string FORMAT Native
42522026-03-10T20:08:11.569ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 24 table_name = fields_string
42532026-03-10T20:08:11.569ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 14 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
42542026-03-10T20:08:11.570ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 14 table_name = fields_u16
42552026-03-10T20:08:11.570ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 26 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
42562026-03-10T20:08:11.572ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 26 table_name = fields_uuid
42572026-03-10T20:08:11.572ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 5 n_rows = 2 sql = INSERT INTO oximeter.measurements_cumulativeu64 FORMAT Native
42582026-03-10T20:08:11.573ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = measurements_cumulativeu64
42592026-03-10T20:08:11.573ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 19 n_rows = 10 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
42602026-03-10T20:08:11.576ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 10 table_name = measurements_histogramu64
42612026-03-10T20:08:11.576ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 13 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
42622026-03-10T20:08:11.577ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 13 table_name = measurements_i64
42632026-03-10T20:08:11.577ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42642026-03-10T20:08:11.643ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
42652026-03-10T20:08:12.029ZTRACtest_retry_backoff (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42662026-03-10T20:08:12.029ZTRACtest_retry_backoff (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42672026-03-10T20:08:12.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42682026-03-10T20:08:12.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42692026-03-10T20:08:12.454ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42702026-03-10T20:08:12.454ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42712026-03-10T20:08:12.454ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42722026-03-10T20:08:12.454ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
42732026-03-10T20:08:13.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42742026-03-10T20:08:13.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42752026-03-10T20:08:13.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
42762026-03-10T20:08:13.029ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
42772026-03-10T20:08:13.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
42782026-03-10T20:08:13.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
42792026-03-10T20:08:13.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
42802026-03-10T20:08:13.033ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42812026-03-10T20:08:13.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
42822026-03-10T20:08:13.442ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
42832026-03-10T20:08:13.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
42842026-03-10T20:08:13.443ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9ecf8b4c-900d-4f40-b02f-3c09394ea4b2 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
42852026-03-10T20:08:13.443ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9ecf8b4c-900d-4f40-b02f-3c09394ea4b2 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
42862026-03-10T20:08:13.443ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9ecf8b4c-900d-4f40-b02f-3c09394ea4b2 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
42872026-03-10T20:08:13.443ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9ecf8b4c-900d-4f40-b02f-3c09394ea4b2 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
42882026-03-10T20:08:13.444ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9ecf8b4c-900d-4f40-b02f-3c09394ea4b2 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
42892026-03-10T20:08:13.447ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3720 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 9ecf8b4c-900d-4f40-b02f-3c09394ea4b2 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
42902026-03-10T20:08:13.447ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ecf8b4c-900d-4f40-b02f-3c09394ea4b2", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:13 GMT"} })
42912026-03-10T20:08:13.447ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
42922026-03-10T20:08:13.447ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 028bb6af-b2dd-489c-b007-c9d41fc314ba uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
42932026-03-10T20:08:13.447ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 028bb6af-b2dd-489c-b007-c9d41fc314ba uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
42942026-03-10T20:08:13.447ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 028bb6af-b2dd-489c-b007-c9d41fc314ba resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
42952026-03-10T20:08:13.447ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 028bb6af-b2dd-489c-b007-c9d41fc314ba roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
42962026-03-10T20:08:13.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 028bb6af-b2dd-489c-b007-c9d41fc314ba resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
42972026-03-10T20:08:13.449ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2160 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 028bb6af-b2dd-489c-b007-c9d41fc314ba response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
42982026-03-10T20:08:13.450ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "028bb6af-b2dd-489c-b007-c9d41fc314ba", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:13 GMT"} })
42992026-03-10T20:08:13.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
43002026-03-10T20:08:13.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43012026-03-10T20:08:13.450ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
43022026-03-10T20:08:13.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43032026-03-10T20:08:13.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43042026-03-10T20:08:13.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43052026-03-10T20:08:13.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43062026-03-10T20:08:13.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43072026-03-10T20:08:13.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43082026-03-10T20:08:14.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43092026-03-10T20:08:14.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43102026-03-10T20:08:14.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
43112026-03-10T20:08:14.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
43122026-03-10T20:08:14.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
43132026-03-10T20:08:14.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
43142026-03-10T20:08:14.034ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
43152026-03-10T20:08:14.034ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43162026-03-10T20:08:14.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
43172026-03-10T20:08:14.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43182026-03-10T20:08:14.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43192026-03-10T20:08:14.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43202026-03-10T20:08:14.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43212026-03-10T20:08:14.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43222026-03-10T20:08:14.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
43232026-03-10T20:08:14.547ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2b c1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 +............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62219 req_id = 0938a4c7-f4ac-4711-b9f0-697196b55910
43242026-03-10T20:08:14.547ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11201,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:62219 req_id = 0938a4c7-f4ac-4711-b9f0-697196b55910
43252026-03-10T20:08:14.548ZDEBGtest_retry_backoff (store): query key key = _mgs._tcp
43262026-03-10T20:08:14.548ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
43272026-03-10T20:08:14.548ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:62219 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 43524, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = 0938a4c7-f4ac-4711-b9f0-697196b55910
43282026-03-10T20:08:14.548ZTRACtest_retry_backoff (dns): buffer buffer = Length: 74 (0x4a) bytes\n0000: f1 98 01 00 00 01 00 00 00 00 00 01 0b 5f 72 65 ............._re\n0010: 70 6f 2d 64 65 70 6f 74 04 5f 74 63 70 0d 63 6f po-depot._tcp.co\n0020: 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 ntrol-plane.oxid\n0030: 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 e.internal..!...\n0040: 00 29 04 d0 00 00 00 00 00 00 .)........ kind = dns peer_addr = [::1]:54393 req_id = 78d0f785-dcc0-459c-8192-9b9f36cf3e29
43292026-03-10T20:08:14.548ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61848,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_repo-depot._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_repo-depot._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 11,\n 95,\n 114,\n 101,\n 112,\n 111,\n 45,\n 100,\n 101,\n 112,\n 111,\n 116,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:54393 req_id = 78d0f785-dcc0-459c-8192-9b9f36cf3e29
43302026-03-10T20:08:14.548ZDEBGtest_retry_backoff (store): query key key = _repo-depot._tcp
43312026-03-10T20:08:14.548ZDEBGtest_retry_backoff (store): query key key = b6d65341-167c-41df-9b5c-41cded99c229.sled
43322026-03-10T20:08:14.548ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:54393 query = LowerQuery { name: LowerName(Name("_repo-depot._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_repo-depot._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_repo-depot._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 47851, target: Name("b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal") }) }] req_id = 78d0f785-dcc0-459c-8192-9b9f36cf3e29
43332026-03-10T20:08:14.548ZTRACtest_retry_backoff (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 78 b8 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 x............_cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:50647 req_id = f03d6421-8e4f-44cb-925e-b5f1864591bf
43342026-03-10T20:08:14.548ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30904,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:50647 req_id = f03d6421-8e4f-44cb-925e-b5f1864591bf
43352026-03-10T20:08:14.548ZDEBGtest_retry_backoff (store): query key key = _crucible-pantry._tcp
43362026-03-10T20:08:14.548ZDEBGtest_retry_backoff (store): query key key = 3b196367-678f-4598-9fb6-cf1a4563fd74.host
43372026-03-10T20:08:14.548ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("3b196367-678f-4598-9fb6-cf1a4563fd74.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:50647 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 36870, target: Name("3b196367-678f-4598-9fb6-cf1a4563fd74.host.control-plane.oxide.internal") }) }] req_id = f03d6421-8e4f-44cb-925e-b5f1864591bf
43382026-03-10T20:08:14.549ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ed b6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60857 req_id = a8c1b345-c2ef-4e9b-88bd-740e40905fbd
43392026-03-10T20:08:14.549ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60854,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:60857 req_id = a8c1b345-c2ef-4e9b-88bd-740e40905fbd
43402026-03-10T20:08:14.549ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
43412026-03-10T20:08:14.549ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60857 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = a8c1b345-c2ef-4e9b-88bd-740e40905fbd
43422026-03-10T20:08:14.549ZTRACtest_retry_backoff (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 04 ba 01 00 00 01 00 00 00 00 00 01 24 62 36 64 ............$b6d\n0010: 36 35 33 34 31 2d 31 36 37 63 2d 34 31 64 66 2d 65341-167c-41df-\n0020: 39 62 35 63 2d 34 31 63 64 65 64 39 39 63 32 32 9b5c-41cded99c22\n0030: 39 04 73 6c 65 64 0d 63 6f 6e 74 72 6f 6c 2d 70 9.sled.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:49295 req_id = 113cce9d-2c7b-4bbd-98ee-8f9002b2b45c
43432026-03-10T20:08:14.549ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1210,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 36,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 115,\n 108,\n 101,\n 100,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:49295 req_id = 113cce9d-2c7b-4bbd-98ee-8f9002b2b45c
43442026-03-10T20:08:14.549ZDEBGtest_retry_backoff (store): query key key = b6d65341-167c-41df-9b5c-41cded99c229.sled
43452026-03-10T20:08:14.549ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49295 query = LowerQuery { name: LowerName(Name("b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal.")), original: Query { name: Name("b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 113cce9d-2c7b-4bbd-98ee-8f9002b2b45c
43462026-03-10T20:08:14.549ZTRACtest_retry_backoff (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 36 c1 01 00 00 01 00 00 00 00 00 01 24 33 62 31 6...........$3b1\n0010: 39 36 33 36 37 2d 36 37 38 66 2d 34 35 39 38 2d 96367-678f-4598-\n0020: 39 66 62 36 2d 63 66 31 61 34 35 36 33 66 64 37 9fb6-cf1a4563fd7\n0030: 34 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 4.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:53804 req_id = edf8d357-a986-4c08-a1a7-7650845d45dd
43472026-03-10T20:08:14.549ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14017,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("3b196367-678f-4598-9fb6-cf1a4563fd74.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("3b196367-678f-4598-9fb6-cf1a4563fd74.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 36,\n 51,\n 98,\n 49,\n 57,\n 54,\n 51,\n 54,\n 55,\n 45,\n 54,\n 55,\n 56,\n 102,\n 45,\n 52,\n 53,\n 57,\n 56,\n 45,\n 57,\n 102,\n 98,\n 54,\n 45,\n 99,\n 102,\n 49,\n 97,\n 52,\n 53,\n 54,\n 51,\n 102,\n 100,\n 55,\n 52,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:53804 req_id = edf8d357-a986-4c08-a1a7-7650845d45dd
43482026-03-10T20:08:14.550ZDEBGtest_retry_backoff (store): query key key = 3b196367-678f-4598-9fb6-cf1a4563fd74.host
43492026-03-10T20:08:14.550ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53804 query = LowerQuery { name: LowerName(Name("3b196367-678f-4598-9fb6-cf1a4563fd74.host.control-plane.oxide.internal.")), original: Query { name: Name("3b196367-678f-4598-9fb6-cf1a4563fd74.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("3b196367-678f-4598-9fb6-cf1a4563fd74.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = edf8d357-a986-4c08-a1a7-7650845d45dd
43502026-03-10T20:08:14.607ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:44507 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
43512026-03-10T20:08:14.608ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:44507 req_id = 9951c6e7-6432-43f5-8bc0-b2a1b2283174 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43522026-03-10T20:08:14.608ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 200 local_addr = [::1]:36870 method = GET remote_addr = [::1]:44507 req_id = 9951c6e7-6432-43f5-8bc0-b2a1b2283174 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43532026-03-10T20:08:15.018ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:46251 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
43542026-03-10T20:08:15.018ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:37016 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
43552026-03-10T20:08:15.018ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:57316 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
43562026-03-10T20:08:15.018ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:35892 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
43572026-03-10T20:08:15.018ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:54085 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
43582026-03-10T20:08:15.018ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:54833 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
43592026-03-10T20:08:15.018ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:62245 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
43602026-03-10T20:08:15.019ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:46251 req_id = 11669c4e-f9db-499d-92ae-f2bb7e63ef0e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43612026-03-10T20:08:15.019ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:37016 req_id = a4646801-5246-4e15-a1c7-e207046a53be sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43622026-03-10T20:08:15.019ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:57316 req_id = a57a2fe4-3e19-4890-9423-02bcfb19607e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43632026-03-10T20:08:15.019ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:35892 req_id = dc39f4ba-d8b8-4d70-bd7d-d39464a0eedd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43642026-03-10T20:08:15.019ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:54085 req_id = c98ed977-eb56-4c84-892b-88c86021306b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43652026-03-10T20:08:15.019ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:54833 req_id = 2f625588-8cf2-43fc-abff-e3d9457f4d7e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43662026-03-10T20:08:15.019ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:62245 req_id = bee016b6-d9fb-4b52-b595-ad9619d9515a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43672026-03-10T20:08:15.019ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 622 local_addr = [::1]:36870 method = GET remote_addr = [::1]:46251 req_id = 11669c4e-f9db-499d-92ae-f2bb7e63ef0e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43682026-03-10T20:08:15.020ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 560 local_addr = [::1]:36870 method = GET remote_addr = [::1]:37016 req_id = a4646801-5246-4e15-a1c7-e207046a53be response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43692026-03-10T20:08:15.020ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 507 local_addr = [::1]:36870 method = GET remote_addr = [::1]:57316 req_id = a57a2fe4-3e19-4890-9423-02bcfb19607e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43702026-03-10T20:08:15.020ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 446 local_addr = [::1]:36870 method = GET remote_addr = [::1]:35892 req_id = dc39f4ba-d8b8-4d70-bd7d-d39464a0eedd response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43712026-03-10T20:08:15.020ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 383 local_addr = [::1]:36870 method = GET remote_addr = [::1]:54085 req_id = c98ed977-eb56-4c84-892b-88c86021306b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43722026-03-10T20:08:15.020ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 323 local_addr = [::1]:36870 method = GET remote_addr = [::1]:54833 req_id = 2f625588-8cf2-43fc-abff-e3d9457f4d7e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43732026-03-10T20:08:15.020ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 269 local_addr = [::1]:36870 method = GET remote_addr = [::1]:62245 req_id = bee016b6-d9fb-4b52-b595-ad9619d9515a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
43742026-03-10T20:08:15.028ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43752026-03-10T20:08:15.028ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43762026-03-10T20:08:15.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
43772026-03-10T20:08:15.029ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
43782026-03-10T20:08:15.035ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
43792026-03-10T20:08:15.035ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
43802026-03-10T20:08:15.036ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
43812026-03-10T20:08:15.036ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43822026-03-10T20:08:15.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
43832026-03-10T20:08:15.443ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
43842026-03-10T20:08:15.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43852026-03-10T20:08:15.444ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 02c2c80b-5562-452f-a40b-e669ec9e491b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43862026-03-10T20:08:15.444ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 02c2c80b-5562-452f-a40b-e669ec9e491b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
43872026-03-10T20:08:15.444ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 02c2c80b-5562-452f-a40b-e669ec9e491b resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
43882026-03-10T20:08:15.444ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 02c2c80b-5562-452f-a40b-e669ec9e491b roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
43892026-03-10T20:08:15.445ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 02c2c80b-5562-452f-a40b-e669ec9e491b resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
43902026-03-10T20:08:15.447ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3161 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 02c2c80b-5562-452f-a40b-e669ec9e491b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43912026-03-10T20:08:15.447ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "02c2c80b-5562-452f-a40b-e669ec9e491b", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:15 GMT"} })
43922026-03-10T20:08:15.447ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43932026-03-10T20:08:15.447ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a119c31f-f687-46af-b4e7-be6e37572155 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43942026-03-10T20:08:15.447ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a119c31f-f687-46af-b4e7-be6e37572155 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
43952026-03-10T20:08:15.448ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a119c31f-f687-46af-b4e7-be6e37572155 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
43962026-03-10T20:08:15.448ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a119c31f-f687-46af-b4e7-be6e37572155 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
43972026-03-10T20:08:15.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a119c31f-f687-46af-b4e7-be6e37572155 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
43982026-03-10T20:08:15.450ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2265 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a119c31f-f687-46af-b4e7-be6e37572155 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43992026-03-10T20:08:15.450ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "a119c31f-f687-46af-b4e7-be6e37572155", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:15 GMT"} })
44002026-03-10T20:08:15.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
44012026-03-10T20:08:15.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44022026-03-10T20:08:15.450ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
44032026-03-10T20:08:15.451ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44042026-03-10T20:08:15.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44052026-03-10T20:08:15.452ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44062026-03-10T20:08:15.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44072026-03-10T20:08:15.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44082026-03-10T20:08:15.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44092026-03-10T20:08:15.565ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = reconfigurator_config_watcher iteration = 3 reason = Timeout
44102026-03-10T20:08:15.566ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig
44112026-03-10T20:08:15.566ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = reconfigurator_config_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44122026-03-10T20:08:15.566ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
44132026-03-10T20:08:15.566ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
44142026-03-10T20:08:15.566ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
44152026-03-10T20:08:15.568ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44162026-03-10T20:08:15.573ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig result = Ok(())
44172026-03-10T20:08:15.573ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
44182026-03-10T20:08:15.573ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
44192026-03-10T20:08:15.573ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
44202026-03-10T20:08:15.575ZDEBGtest_retry_backoff (BackgroundTasks): reconfigurator config load complete background_task = reconfigurator_config_watcher switches = Loaded(ReconfiguratorConfigView { version: 1, config: ReconfiguratorConfig { planner_enabled: false, planner_config: PlannerConfig { add_zones_with_mupdate_override: false }, tuf_repo_pruner_enabled: true }, time_modified: 2026-03-10T20:08:05.458980Z })
44212026-03-10T20:08:15.575ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = reconfigurator_config_watcher elapsed = 9.559595ms iteration = 3 status = {"config_updated":false}
44222026-03-10T20:08:16.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44232026-03-10T20:08:16.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44242026-03-10T20:08:16.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
44252026-03-10T20:08:16.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
44262026-03-10T20:08:16.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
44272026-03-10T20:08:16.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
44282026-03-10T20:08:16.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
44292026-03-10T20:08:16.033ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44302026-03-10T20:08:16.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
44312026-03-10T20:08:16.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44322026-03-10T20:08:16.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44332026-03-10T20:08:16.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44342026-03-10T20:08:16.454ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44352026-03-10T20:08:16.454ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44362026-03-10T20:08:16.454ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44372026-03-10T20:08:17.030ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44382026-03-10T20:08:17.030ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44392026-03-10T20:08:17.030ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
44402026-03-10T20:08:17.031ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
44412026-03-10T20:08:17.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
44422026-03-10T20:08:17.033ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
44432026-03-10T20:08:17.035ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
44442026-03-10T20:08:17.035ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44452026-03-10T20:08:17.131ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
44462026-03-10T20:08:17.443ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
44472026-03-10T20:08:17.444ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44482026-03-10T20:08:17.444ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 6667b9af-94aa-470e-b85e-53aab0e324bd uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44492026-03-10T20:08:17.444ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 6667b9af-94aa-470e-b85e-53aab0e324bd uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
44502026-03-10T20:08:17.444ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 6667b9af-94aa-470e-b85e-53aab0e324bd resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
44512026-03-10T20:08:17.444ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 6667b9af-94aa-470e-b85e-53aab0e324bd roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
44522026-03-10T20:08:17.445ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 6667b9af-94aa-470e-b85e-53aab0e324bd resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
44532026-03-10T20:08:17.447ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3392 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 6667b9af-94aa-470e-b85e-53aab0e324bd response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44542026-03-10T20:08:17.447ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6667b9af-94aa-470e-b85e-53aab0e324bd", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:17 GMT"} })
44552026-03-10T20:08:17.448ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44562026-03-10T20:08:17.448ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = f6417004-ac58-4903-932f-ff094c7555fe uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44572026-03-10T20:08:17.448ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = f6417004-ac58-4903-932f-ff094c7555fe uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
44582026-03-10T20:08:17.448ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = f6417004-ac58-4903-932f-ff094c7555fe resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
44592026-03-10T20:08:17.448ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = f6417004-ac58-4903-932f-ff094c7555fe roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
44602026-03-10T20:08:17.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = f6417004-ac58-4903-932f-ff094c7555fe resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
44612026-03-10T20:08:17.451ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2850 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = f6417004-ac58-4903-932f-ff094c7555fe response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44622026-03-10T20:08:17.451ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6417004-ac58-4903-932f-ff094c7555fe", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:17 GMT"} })
44632026-03-10T20:08:17.451ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
44642026-03-10T20:08:17.451ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44652026-03-10T20:08:17.451ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
44662026-03-10T20:08:17.451ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44672026-03-10T20:08:17.451ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44682026-03-10T20:08:17.452ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44692026-03-10T20:08:17.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44702026-03-10T20:08:17.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44712026-03-10T20:08:17.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44722026-03-10T20:08:18.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44732026-03-10T20:08:18.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44742026-03-10T20:08:18.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
44752026-03-10T20:08:18.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
44762026-03-10T20:08:18.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
44772026-03-10T20:08:18.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
44782026-03-10T20:08:18.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
44792026-03-10T20:08:18.033ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44802026-03-10T20:08:18.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
44812026-03-10T20:08:18.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44822026-03-10T20:08:18.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44832026-03-10T20:08:18.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44842026-03-10T20:08:18.454ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44852026-03-10T20:08:18.454ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44862026-03-10T20:08:18.454ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
44872026-03-10T20:08:19.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44882026-03-10T20:08:19.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44892026-03-10T20:08:19.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
44902026-03-10T20:08:19.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
44912026-03-10T20:08:19.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
44922026-03-10T20:08:19.033ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
44932026-03-10T20:08:19.035ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
44942026-03-10T20:08:19.035ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44952026-03-10T20:08:19.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
44962026-03-10T20:08:19.443ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
44972026-03-10T20:08:19.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44982026-03-10T20:08:19.444ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 66322e23-fa67-4167-8da3-33cddf8a7de6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44992026-03-10T20:08:19.444ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 66322e23-fa67-4167-8da3-33cddf8a7de6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
45002026-03-10T20:08:19.444ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 66322e23-fa67-4167-8da3-33cddf8a7de6 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
45012026-03-10T20:08:19.444ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 66322e23-fa67-4167-8da3-33cddf8a7de6 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
45022026-03-10T20:08:19.444ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 66322e23-fa67-4167-8da3-33cddf8a7de6 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
45032026-03-10T20:08:19.447ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2968 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 66322e23-fa67-4167-8da3-33cddf8a7de6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
45042026-03-10T20:08:19.447ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "66322e23-fa67-4167-8da3-33cddf8a7de6", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:19 GMT"} })
45052026-03-10T20:08:19.447ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45062026-03-10T20:08:19.447ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 158634b4-36fe-4154-97f2-d98298e24c0b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45072026-03-10T20:08:19.447ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 158634b4-36fe-4154-97f2-d98298e24c0b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
45082026-03-10T20:08:19.447ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 158634b4-36fe-4154-97f2-d98298e24c0b resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
45092026-03-10T20:08:19.447ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 158634b4-36fe-4154-97f2-d98298e24c0b roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
45102026-03-10T20:08:19.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 158634b4-36fe-4154-97f2-d98298e24c0b resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
45112026-03-10T20:08:19.449ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2012 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 158634b4-36fe-4154-97f2-d98298e24c0b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
45122026-03-10T20:08:19.449ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "158634b4-36fe-4154-97f2-d98298e24c0b", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:19 GMT"} })
45132026-03-10T20:08:19.449ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
45142026-03-10T20:08:19.449ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
45152026-03-10T20:08:19.449ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
45162026-03-10T20:08:19.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45172026-03-10T20:08:19.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45182026-03-10T20:08:19.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45192026-03-10T20:08:19.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45202026-03-10T20:08:19.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45212026-03-10T20:08:19.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45222026-03-10T20:08:20.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
45232026-03-10T20:08:20.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
45242026-03-10T20:08:20.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
45252026-03-10T20:08:20.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
45262026-03-10T20:08:20.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
45272026-03-10T20:08:20.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
45282026-03-10T20:08:20.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
45292026-03-10T20:08:20.033ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
45302026-03-10T20:08:20.129ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
45312026-03-10T20:08:20.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45322026-03-10T20:08:20.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45332026-03-10T20:08:20.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45342026-03-10T20:08:20.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45352026-03-10T20:08:20.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45362026-03-10T20:08:20.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
45372026-03-10T20:08:20.561ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = inventory_loader iteration = 3 reason = Timeout
45382026-03-10T20:08:20.561ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Inventory
45392026-03-10T20:08:20.561ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = inventory_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45402026-03-10T20:08:20.561ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database
45412026-03-10T20:08:20.561ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {} }
45422026-03-10T20:08:20.561ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database result = Ok(())
45432026-03-10T20:08:20.562ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = reconfigurator_config_watcher iteration = 4 reason = Timeout
45442026-03-10T20:08:20.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig
45452026-03-10T20:08:20.562ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = reconfigurator_config_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45462026-03-10T20:08:20.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
45472026-03-10T20:08:20.562ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
45482026-03-10T20:08:20.563ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
45492026-03-10T20:08:20.564ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45502026-03-10T20:08:20.568ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig result = Ok(())
45512026-03-10T20:08:20.568ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
45522026-03-10T20:08:20.568ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
45532026-03-10T20:08:20.569ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
45542026-03-10T20:08:20.569ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45552026-03-10T20:08:20.573ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Inventory result = Ok(())
45562026-03-10T20:08:20.573ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database
45572026-03-10T20:08:20.573ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {} }
45582026-03-10T20:08:20.573ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database result = Ok(())
45592026-03-10T20:08:20.575ZDEBGtest_retry_backoff (BackgroundTasks): reconfigurator config load complete background_task = reconfigurator_config_watcher switches = Loaded(ReconfiguratorConfigView { version: 1, config: ReconfiguratorConfig { planner_enabled: false, planner_config: PlannerConfig { add_zones_with_mupdate_override: false }, tuf_repo_pruner_enabled: true }, time_modified: 2026-03-10T20:08:05.458980Z })
45602026-03-10T20:08:20.575ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = reconfigurator_config_watcher elapsed = 12.41224ms iteration = 4 status = {"config_updated":false}
45612026-03-10T20:08:20.575ZDEBGtest_retry_backoff (BackgroundTasks): latest inventory collection is unchanged background_task = inventory_loader original_id = 217d854a-1530-402d-b94f-a3562c53f830 original_time_started = 2026-03-10 20:08:08.140360 UTC
45622026-03-10T20:08:20.575ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = inventory_loader elapsed = 14.219626ms iteration = 3 status = {"Loaded":{"collection_id":"217d854a-1530-402d-b94f-a3562c53f830","time_loaded":"2026-03-10T20:08:20.561217042Z","time_started":"2026-03-10T20:08:08.140360Z"}}
45632026-03-10T20:08:20.604ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = fm_sitrep_loader iteration = 2 reason = Timeout
45642026-03-10T20:08:20.604ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Fleet
45652026-03-10T20:08:20.604ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = fm_sitrep_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45662026-03-10T20:08:20.604ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database
45672026-03-10T20:08:20.604ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_loader roles = RoleSet { roles: {} }
45682026-03-10T20:08:20.605ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database result = Ok(())
45692026-03-10T20:08:20.606ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = fm_sitrep_gc iteration = 2 reason = Timeout
45702026-03-10T20:08:20.606ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
45712026-03-10T20:08:20.606ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
45722026-03-10T20:08:20.606ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
45732026-03-10T20:08:20.607ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet
45742026-03-10T20:08:20.607ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = fm_sitrep_gc resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45752026-03-10T20:08:20.607ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
45762026-03-10T20:08:20.607ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
45772026-03-10T20:08:20.607ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
45782026-03-10T20:08:20.607ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45792026-03-10T20:08:20.610ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Fleet result = Ok(())
45802026-03-10T20:08:20.611ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database
45812026-03-10T20:08:20.611ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_loader roles = RoleSet { roles: {} }
45822026-03-10T20:08:20.611ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database result = Ok(())
45832026-03-10T20:08:20.612ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45842026-03-10T20:08:20.614ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet result = Ok(())
45852026-03-10T20:08:20.614ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = fm_sitrep_loader elapsed = 9.679016ms iteration = 2 status = "NoSitrep"
45862026-03-10T20:08:20.617ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
45872026-03-10T20:08:20.617ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
45882026-03-10T20:08:20.617ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
45892026-03-10T20:08:20.619ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet
45902026-03-10T20:08:20.619ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = fm_sitrep_gc resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45912026-03-10T20:08:20.619ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
45922026-03-10T20:08:20.619ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
45932026-03-10T20:08:20.620ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
45942026-03-10T20:08:20.622ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45952026-03-10T20:08:20.624ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet result = Ok(())
45962026-03-10T20:08:20.633ZINFOtest_retry_backoff (BackgroundTasks): deleted 0 of 0 sitreps sitreps background_task = fm_sitrep_gc case_ereports_deleted = 0 cases_deleted = 0 ids = [] sitreps_deleted = 0
45972026-03-10T20:08:20.633ZTRACtest_retry_backoff (BackgroundTasks): all 0 orphaned sitreps in this batch have already been deleted background_task = fm_sitrep_gc
45982026-03-10T20:08:20.633ZINFOtest_retry_backoff (BackgroundTasks): sitrep garbage collection found 0 orphaned sitreps, deleted 0 background_task = fm_sitrep_gc
45992026-03-10T20:08:20.633ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = fm_sitrep_gc elapsed = 27.53785ms iteration = 2 status = {"errors":[],"orphaned_sitreps_deleted":0,"orphaned_sitreps_found":0}
46002026-03-10T20:08:21.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46012026-03-10T20:08:21.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46022026-03-10T20:08:21.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
46032026-03-10T20:08:21.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
46042026-03-10T20:08:21.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
46052026-03-10T20:08:21.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
46062026-03-10T20:08:21.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
46072026-03-10T20:08:21.033ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46082026-03-10T20:08:21.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
46092026-03-10T20:08:21.443ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
46102026-03-10T20:08:21.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46112026-03-10T20:08:21.443ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 8621a19c-e8c6-4f70-a888-cc104087b40e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46122026-03-10T20:08:21.443ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 8621a19c-e8c6-4f70-a888-cc104087b40e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
46132026-03-10T20:08:21.443ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 8621a19c-e8c6-4f70-a888-cc104087b40e resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
46142026-03-10T20:08:21.444ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 8621a19c-e8c6-4f70-a888-cc104087b40e roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
46152026-03-10T20:08:21.444ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 8621a19c-e8c6-4f70-a888-cc104087b40e resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
46162026-03-10T20:08:21.447ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3699 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 8621a19c-e8c6-4f70-a888-cc104087b40e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46172026-03-10T20:08:21.447ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "8621a19c-e8c6-4f70-a888-cc104087b40e", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:21 GMT"} })
46182026-03-10T20:08:21.447ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46192026-03-10T20:08:21.448ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7c2ca25a-3b3e-449a-8cfd-e2967d0a0f7c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46202026-03-10T20:08:21.448ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7c2ca25a-3b3e-449a-8cfd-e2967d0a0f7c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
46212026-03-10T20:08:21.448ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7c2ca25a-3b3e-449a-8cfd-e2967d0a0f7c resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
46222026-03-10T20:08:21.448ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7c2ca25a-3b3e-449a-8cfd-e2967d0a0f7c roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
46232026-03-10T20:08:21.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7c2ca25a-3b3e-449a-8cfd-e2967d0a0f7c resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
46242026-03-10T20:08:21.450ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2272 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7c2ca25a-3b3e-449a-8cfd-e2967d0a0f7c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46252026-03-10T20:08:21.450ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c2ca25a-3b3e-449a-8cfd-e2967d0a0f7c", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:21 GMT"} })
46262026-03-10T20:08:21.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
46272026-03-10T20:08:21.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46282026-03-10T20:08:21.450ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
46292026-03-10T20:08:21.451ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
46302026-03-10T20:08:21.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46312026-03-10T20:08:21.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
46322026-03-10T20:08:21.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46332026-03-10T20:08:21.452ZTRACtest_retry_backoff (dropshot): incoming request local_addr = 127.0.0.1:33574 method = GET remote_addr = 127.0.0.1:51522 req_id = ba5219e6-fcba-4d81-9c6f-76cff7fb1cd8 uri = /913233fe-92a8-4635-9572-183f495429c4
46342026-03-10T20:08:21.458ZINFOtest_retry_backoff (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867 latency_us = 5944 local_addr = 127.0.0.1:33574 method = GET remote_addr = 127.0.0.1:51522 req_id = ba5219e6-fcba-4d81-9c6f-76cff7fb1cd8 response_code = 200 uri = /913233fe-92a8-4635-9572-183f495429c4
46352026-03-10T20:08:21.458ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
46362026-03-10T20:08:21.458ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46372026-03-10T20:08:21.459ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46382026-03-10T20:08:21.459ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46392026-03-10T20:08:21.459ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46402026-03-10T20:08:21.464ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
46412026-03-10T20:08:21.465ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
46422026-03-10T20:08:21.465ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
46432026-03-10T20:08:21.465ZDEBGtest_retry_backoff (results-sink): inserting 11 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46442026-03-10T20:08:21.465ZDEBGtest_retry_backoff (results-sink-cluster): inserting 11 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46452026-03-10T20:08:21.465ZDEBGtest_retry_backoff (clickhouse-client): unrolling 11 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
46462026-03-10T20:08:21.470ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 22 sql = INSERT INTO oximeter.fields_string FORMAT Native
46472026-03-10T20:08:21.472ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 22 table_name = fields_string
46482026-03-10T20:08:21.472ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
46492026-03-10T20:08:21.473ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 10 table_name = fields_u16
46502026-03-10T20:08:21.473ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
46512026-03-10T20:08:21.475ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 10 table_name = fields_uuid
46522026-03-10T20:08:21.475ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 19 n_rows = 10 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
46532026-03-10T20:08:21.477ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 10 table_name = measurements_histogramu64
46542026-03-10T20:08:21.477ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
46552026-03-10T20:08:21.478ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
46562026-03-10T20:08:21.478ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46572026-03-10T20:08:21.566ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
46582026-03-10T20:08:22.029ZTRACtest_retry_backoff (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46592026-03-10T20:08:22.029ZTRACtest_retry_backoff (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46602026-03-10T20:08:22.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46612026-03-10T20:08:22.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46622026-03-10T20:08:22.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46632026-03-10T20:08:22.454ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46642026-03-10T20:08:22.454ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46652026-03-10T20:08:22.454ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46662026-03-10T20:08:23.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46672026-03-10T20:08:23.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46682026-03-10T20:08:23.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
46692026-03-10T20:08:23.029ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
46702026-03-10T20:08:23.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
46712026-03-10T20:08:23.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
46722026-03-10T20:08:23.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
46732026-03-10T20:08:23.034ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
46742026-03-10T20:08:23.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
46752026-03-10T20:08:23.443ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
46762026-03-10T20:08:23.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46772026-03-10T20:08:23.444ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 62114acb-835a-4547-aad9-9ce6244bc938 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46782026-03-10T20:08:23.444ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 62114acb-835a-4547-aad9-9ce6244bc938 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
46792026-03-10T20:08:23.444ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 62114acb-835a-4547-aad9-9ce6244bc938 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
46802026-03-10T20:08:23.444ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 62114acb-835a-4547-aad9-9ce6244bc938 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
46812026-03-10T20:08:23.444ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 62114acb-835a-4547-aad9-9ce6244bc938 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
46822026-03-10T20:08:23.447ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3436 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 62114acb-835a-4547-aad9-9ce6244bc938 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46832026-03-10T20:08:23.447ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "62114acb-835a-4547-aad9-9ce6244bc938", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:23 GMT"} })
46842026-03-10T20:08:23.447ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46852026-03-10T20:08:23.448ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 532e57b2-4a97-454b-942a-4027aa44e78e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46862026-03-10T20:08:23.448ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 532e57b2-4a97-454b-942a-4027aa44e78e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
46872026-03-10T20:08:23.448ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 532e57b2-4a97-454b-942a-4027aa44e78e resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
46882026-03-10T20:08:23.448ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 532e57b2-4a97-454b-942a-4027aa44e78e roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
46892026-03-10T20:08:23.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 532e57b2-4a97-454b-942a-4027aa44e78e resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
46902026-03-10T20:08:23.450ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2320 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 532e57b2-4a97-454b-942a-4027aa44e78e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46912026-03-10T20:08:23.450ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "532e57b2-4a97-454b-942a-4027aa44e78e", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:23 GMT"} })
46922026-03-10T20:08:23.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
46932026-03-10T20:08:23.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46942026-03-10T20:08:23.450ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
46952026-03-10T20:08:23.451ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46962026-03-10T20:08:23.451ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46972026-03-10T20:08:23.452ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46982026-03-10T20:08:23.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
46992026-03-10T20:08:23.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47002026-03-10T20:08:23.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47012026-03-10T20:08:24.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47022026-03-10T20:08:24.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47032026-03-10T20:08:24.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
47042026-03-10T20:08:24.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
47052026-03-10T20:08:24.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
47062026-03-10T20:08:24.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
47072026-03-10T20:08:24.034ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
47082026-03-10T20:08:24.034ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47092026-03-10T20:08:24.131ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
47102026-03-10T20:08:24.451ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47112026-03-10T20:08:24.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47122026-03-10T20:08:24.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47132026-03-10T20:08:24.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47142026-03-10T20:08:24.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47152026-03-10T20:08:24.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47162026-03-10T20:08:25.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47172026-03-10T20:08:25.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47182026-03-10T20:08:25.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
47192026-03-10T20:08:25.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
47202026-03-10T20:08:25.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
47212026-03-10T20:08:25.033ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
47222026-03-10T20:08:25.034ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
47232026-03-10T20:08:25.034ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47242026-03-10T20:08:25.131ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
47252026-03-10T20:08:25.416ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
47262026-03-10T20:08:25.416ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e386e163-c45b-409e-8ac3-44c52b874aa4 uri = /bgtasks/view/webhook_deliverator
47272026-03-10T20:08:25.416ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e386e163-c45b-409e-8ac3-44c52b874aa4 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47282026-03-10T20:08:25.416ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e386e163-c45b-409e-8ac3-44c52b874aa4 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47292026-03-10T20:08:25.417ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e386e163-c45b-409e-8ac3-44c52b874aa4 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47302026-03-10T20:08:25.417ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e386e163-c45b-409e-8ac3-44c52b874aa4 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47312026-03-10T20:08:25.417ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e386e163-c45b-409e-8ac3-44c52b874aa4 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47322026-03-10T20:08:25.420ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e386e163-c45b-409e-8ac3-44c52b874aa4 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47332026-03-10T20:08:25.423ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e386e163-c45b-409e-8ac3-44c52b874aa4 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47342026-03-10T20:08:25.423ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 7021 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e386e163-c45b-409e-8ac3-44c52b874aa4 response_code = 200 uri = /bgtasks/view/webhook_deliverator
47352026-03-10T20:08:25.424ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
47362026-03-10T20:08:25.424ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:50070/bgtasks/activate
47372026-03-10T20:08:25.424ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 0f84c326-bd05-487e-b955-895f45f42c18 uri = /bgtasks/activate
47382026-03-10T20:08:25.424ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 0f84c326-bd05-487e-b955-895f45f42c18 resource = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
47392026-03-10T20:08:25.424ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 0f84c326-bd05-487e-b955-895f45f42c18 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
47402026-03-10T20:08:25.424ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 0f84c326-bd05-487e-b955-895f45f42c18 resource = Database type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
47412026-03-10T20:08:25.424ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 0f84c326-bd05-487e-b955-895f45f42c18 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
47422026-03-10T20:08:25.425ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 0f84c326-bd05-487e-b955-895f45f42c18 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
47432026-03-10T20:08:25.427ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 0f84c326-bd05-487e-b955-895f45f42c18 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
47442026-03-10T20:08:25.429ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 0f84c326-bd05-487e-b955-895f45f42c18 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
47452026-03-10T20:08:25.429ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = webhook_deliverator iteration = 5 reason = Signaled
47462026-03-10T20:08:25.429ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database
47472026-03-10T20:08:25.429ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
47482026-03-10T20:08:25.430ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(())
47492026-03-10T20:08:25.430ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 6362 local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 0f84c326-bd05-487e-b955-895f45f42c18 response_code = 204 uri = /bgtasks/activate
47502026-03-10T20:08:25.431ZINFOtest_retry_backoff (lockstep client test context): client received response status = 204
47512026-03-10T20:08:25.431ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
47522026-03-10T20:08:25.431ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 2e574dfd-0de3-46b0-8e41-96919920ed70 uri = /bgtasks/view/webhook_deliverator
47532026-03-10T20:08:25.431ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 2e574dfd-0de3-46b0-8e41-96919920ed70 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47542026-03-10T20:08:25.431ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 2e574dfd-0de3-46b0-8e41-96919920ed70 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47552026-03-10T20:08:25.431ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 2e574dfd-0de3-46b0-8e41-96919920ed70 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47562026-03-10T20:08:25.431ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 2e574dfd-0de3-46b0-8e41-96919920ed70 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47572026-03-10T20:08:25.431ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 2e574dfd-0de3-46b0-8e41-96919920ed70 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47582026-03-10T20:08:25.433ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 2e574dfd-0de3-46b0-8e41-96919920ed70 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47592026-03-10T20:08:25.436ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 2e574dfd-0de3-46b0-8e41-96919920ed70 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
47602026-03-10T20:08:25.437ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5832 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 2e574dfd-0de3-46b0-8e41-96919920ed70 response_code = 200 uri = /bgtasks/view/webhook_deliverator
47612026-03-10T20:08:25.437ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
47622026-03-10T20:08:25.438ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47632026-03-10T20:08:25.438ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47642026-03-10T20:08:25.439ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47652026-03-10T20:08:25.441ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47662026-03-10T20:08:25.441ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47672026-03-10T20:08:25.442ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47682026-03-10T20:08:25.442ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
47692026-03-10T20:08:25.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
47702026-03-10T20:08:25.443ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 131456b5-81b6-435a-b45b-e28da906f6a8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
47712026-03-10T20:08:25.443ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 131456b5-81b6-435a-b45b-e28da906f6a8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
47722026-03-10T20:08:25.443ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 131456b5-81b6-435a-b45b-e28da906f6a8 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
47732026-03-10T20:08:25.443ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 131456b5-81b6-435a-b45b-e28da906f6a8 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
47742026-03-10T20:08:25.444ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 131456b5-81b6-435a-b45b-e28da906f6a8 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
47752026-03-10T20:08:25.445ZTRACtest_retry_backoff (BackgroundTasks): webhook event delivery attempt started alert_class = test.foo alert_id = 5b38ce09-0fe3-4310-a5b1-e7643a39c1de attempt = 2 background_task = webhook_deliverator delivery_id = 13901bf0-04f6-4b8c-a2c1-278327927fd7 rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47762026-03-10T20:08:25.446ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2636 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 131456b5-81b6-435a-b45b-e28da906f6a8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
47772026-03-10T20:08:25.446ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "131456b5-81b6-435a-b45b-e28da906f6a8", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:25 GMT"} })
47782026-03-10T20:08:25.446ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
47792026-03-10T20:08:25.446ZWARNtest_retry_backoff (BackgroundTasks): webhook receiver endpoint returned an HTTP error alert_class = test.foo alert_id = 5b38ce09-0fe3-4310-a5b1-e7643a39c1de background_task = webhook_deliverator delivery_id = 13901bf0-04f6-4b8c-a2c1-278327927fd7 delivery_trigger = alert response_duration = 1.151697ms response_status = 503 rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47802026-03-10T20:08:25.446ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47812026-03-10T20:08:25.446ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47822026-03-10T20:08:25.447ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
47832026-03-10T20:08:25.448ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4c96d44a-ac39-4d5b-a168-4f3ae6966655 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
47842026-03-10T20:08:25.448ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4c96d44a-ac39-4d5b-a168-4f3ae6966655 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
47852026-03-10T20:08:25.448ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4c96d44a-ac39-4d5b-a168-4f3ae6966655 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
47862026-03-10T20:08:25.448ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4c96d44a-ac39-4d5b-a168-4f3ae6966655 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
47872026-03-10T20:08:25.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4c96d44a-ac39-4d5b-a168-4f3ae6966655 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
47882026-03-10T20:08:25.455ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47892026-03-10T20:08:25.455ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47902026-03-10T20:08:25.456ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47912026-03-10T20:08:25.456ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47922026-03-10T20:08:25.456ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47932026-03-10T20:08:25.457ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
47942026-03-10T20:08:25.457ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 8955 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4c96d44a-ac39-4d5b-a168-4f3ae6966655 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
47952026-03-10T20:08:25.457ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c96d44a-ac39-4d5b-a168-4f3ae6966655", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:25 GMT"} })
47962026-03-10T20:08:25.457ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
47972026-03-10T20:08:25.457ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47982026-03-10T20:08:25.457ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
47992026-03-10T20:08:25.489ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
48002026-03-10T20:08:25.489ZINFOtest_retry_backoff (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator num_receivers = 1
48012026-03-10T20:08:25.489ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 59.634565ms iteration = 5 status = {"by_rx":{"2b6fbbc6-d38e-4d46-a5cd-293e75593af0":{"already_delivered":0,"delivered_ok":0,"delivery_errors":{},"error":null,"failed_deliveries":[{"alert_id":"5b38ce09-0fe3-4310-a5b1-e7643a39c1de","attempt":2,"delivery_id":"13901bf0-04f6-4b8c-a2c1-278327927fd7","response_duration":[0,1151697],"response_status":503,"result":"failed_http_error"}],"in_progress":0,"ready":1}},"error":null}
48022026-03-10T20:08:25.489ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 54411540-c2b1-4a0d-9495-75075bd4dd94 uri = /bgtasks/view/webhook_deliverator
48032026-03-10T20:08:25.489ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 54411540-c2b1-4a0d-9495-75075bd4dd94 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48042026-03-10T20:08:25.489ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 54411540-c2b1-4a0d-9495-75075bd4dd94 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48052026-03-10T20:08:25.489ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 54411540-c2b1-4a0d-9495-75075bd4dd94 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48062026-03-10T20:08:25.489ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 54411540-c2b1-4a0d-9495-75075bd4dd94 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48072026-03-10T20:08:25.490ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 54411540-c2b1-4a0d-9495-75075bd4dd94 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48082026-03-10T20:08:25.492ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 54411540-c2b1-4a0d-9495-75075bd4dd94 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48092026-03-10T20:08:25.495ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 54411540-c2b1-4a0d-9495-75075bd4dd94 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48102026-03-10T20:08:25.495ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 6140 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 54411540-c2b1-4a0d-9495-75075bd4dd94 response_code = 200 uri = /bgtasks/view/webhook_deliverator
48112026-03-10T20:08:25.495ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
48122026-03-10T20:08:25.496ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
48132026-03-10T20:08:25.496ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 7f086480-c687-404d-beb0-63f54ae16d3f uri = /bgtasks/view/webhook_deliverator
48142026-03-10T20:08:25.497ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 7f086480-c687-404d-beb0-63f54ae16d3f resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48152026-03-10T20:08:25.497ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 7f086480-c687-404d-beb0-63f54ae16d3f resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48162026-03-10T20:08:25.497ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 7f086480-c687-404d-beb0-63f54ae16d3f resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48172026-03-10T20:08:25.497ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 7f086480-c687-404d-beb0-63f54ae16d3f roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48182026-03-10T20:08:25.497ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 7f086480-c687-404d-beb0-63f54ae16d3f resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48192026-03-10T20:08:25.499ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 7f086480-c687-404d-beb0-63f54ae16d3f roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48202026-03-10T20:08:25.502ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 7f086480-c687-404d-beb0-63f54ae16d3f resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48212026-03-10T20:08:25.502ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5743 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 7f086480-c687-404d-beb0-63f54ae16d3f response_code = 200 uri = /bgtasks/view/webhook_deliverator
48222026-03-10T20:08:25.502ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
48232026-03-10T20:08:25.503ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:50070/bgtasks/activate
48242026-03-10T20:08:25.503ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = ca8f688a-c825-421a-b27e-c8d0b65ee150 uri = /bgtasks/activate
48252026-03-10T20:08:25.503ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = ca8f688a-c825-421a-b27e-c8d0b65ee150 resource = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
48262026-03-10T20:08:25.503ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = ca8f688a-c825-421a-b27e-c8d0b65ee150 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
48272026-03-10T20:08:25.503ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = ca8f688a-c825-421a-b27e-c8d0b65ee150 resource = Database type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
48282026-03-10T20:08:25.503ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = ca8f688a-c825-421a-b27e-c8d0b65ee150 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
48292026-03-10T20:08:25.504ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = ca8f688a-c825-421a-b27e-c8d0b65ee150 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
48302026-03-10T20:08:25.505ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = ca8f688a-c825-421a-b27e-c8d0b65ee150 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
48312026-03-10T20:08:25.507ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = ca8f688a-c825-421a-b27e-c8d0b65ee150 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
48322026-03-10T20:08:25.507ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = webhook_deliverator iteration = 6 reason = Signaled
48332026-03-10T20:08:25.507ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database
48342026-03-10T20:08:25.507ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
48352026-03-10T20:08:25.508ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(())
48362026-03-10T20:08:25.508ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5670 local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = ca8f688a-c825-421a-b27e-c8d0b65ee150 response_code = 204 uri = /bgtasks/activate
48372026-03-10T20:08:25.509ZINFOtest_retry_backoff (lockstep client test context): client received response status = 204
48382026-03-10T20:08:25.509ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
48392026-03-10T20:08:25.509ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e372d2d0-48f9-4eed-a1f4-c6835ac6ba36 uri = /bgtasks/view/webhook_deliverator
48402026-03-10T20:08:25.509ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e372d2d0-48f9-4eed-a1f4-c6835ac6ba36 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48412026-03-10T20:08:25.509ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e372d2d0-48f9-4eed-a1f4-c6835ac6ba36 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48422026-03-10T20:08:25.509ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e372d2d0-48f9-4eed-a1f4-c6835ac6ba36 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48432026-03-10T20:08:25.509ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e372d2d0-48f9-4eed-a1f4-c6835ac6ba36 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48442026-03-10T20:08:25.509ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e372d2d0-48f9-4eed-a1f4-c6835ac6ba36 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48452026-03-10T20:08:25.511ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e372d2d0-48f9-4eed-a1f4-c6835ac6ba36 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48462026-03-10T20:08:25.514ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e372d2d0-48f9-4eed-a1f4-c6835ac6ba36 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48472026-03-10T20:08:25.514ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5434 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = e372d2d0-48f9-4eed-a1f4-c6835ac6ba36 response_code = 200 uri = /bgtasks/view/webhook_deliverator
48482026-03-10T20:08:25.514ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
48492026-03-10T20:08:25.517ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
48502026-03-10T20:08:25.517ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
48512026-03-10T20:08:25.517ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
48522026-03-10T20:08:25.520ZINFOtest_retry_backoff (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator num_receivers = 1
48532026-03-10T20:08:25.520ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 12.736434ms iteration = 6 status = {"by_rx":{"2b6fbbc6-d38e-4d46-a5cd-293e75593af0":{"already_delivered":0,"delivered_ok":0,"delivery_errors":{},"error":null,"failed_deliveries":[],"in_progress":0,"ready":0}},"error":null}
48542026-03-10T20:08:25.562ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = reconfigurator_config_watcher iteration = 5 reason = Timeout
48552026-03-10T20:08:25.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig
48562026-03-10T20:08:25.562ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = reconfigurator_config_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48572026-03-10T20:08:25.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
48582026-03-10T20:08:25.562ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
48592026-03-10T20:08:25.563ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
48602026-03-10T20:08:25.565ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48612026-03-10T20:08:25.569ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig result = Ok(())
48622026-03-10T20:08:25.569ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
48632026-03-10T20:08:25.569ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
48642026-03-10T20:08:25.569ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
48652026-03-10T20:08:25.570ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
48662026-03-10T20:08:25.570ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = c228180b-54bc-47f5-a36a-813fa8861d21 uri = /bgtasks/view/webhook_deliverator
48672026-03-10T20:08:25.570ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = c228180b-54bc-47f5-a36a-813fa8861d21 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48682026-03-10T20:08:25.570ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = c228180b-54bc-47f5-a36a-813fa8861d21 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48692026-03-10T20:08:25.570ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = c228180b-54bc-47f5-a36a-813fa8861d21 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48702026-03-10T20:08:25.570ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = c228180b-54bc-47f5-a36a-813fa8861d21 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48712026-03-10T20:08:25.571ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = c228180b-54bc-47f5-a36a-813fa8861d21 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48722026-03-10T20:08:25.573ZDEBGtest_retry_backoff (BackgroundTasks): reconfigurator config load complete background_task = reconfigurator_config_watcher switches = Loaded(ReconfiguratorConfigView { version: 1, config: ReconfiguratorConfig { planner_enabled: false, planner_config: PlannerConfig { add_zones_with_mupdate_override: false }, tuf_repo_pruner_enabled: true }, time_modified: 2026-03-10T20:08:05.458980Z })
48732026-03-10T20:08:25.573ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = reconfigurator_config_watcher elapsed = 10.813348ms iteration = 5 status = {"config_updated":false}
48742026-03-10T20:08:25.573ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = c228180b-54bc-47f5-a36a-813fa8861d21 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48752026-03-10T20:08:25.576ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = c228180b-54bc-47f5-a36a-813fa8861d21 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
48762026-03-10T20:08:25.576ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 6331 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = c228180b-54bc-47f5-a36a-813fa8861d21 response_code = 200 uri = /bgtasks/view/webhook_deliverator
48772026-03-10T20:08:25.577ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
48782026-03-10T20:08:25.577ZINFOtest_retry_backoff (external client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:46842/v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
48792026-03-10T20:08:25.577ZTRACtest_retry_backoff (dropshot_external): incoming request local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
48802026-03-10T20:08:25.577ZDEBGtest_retry_backoff (dropshot_external): determined request API version local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10 version = 2026030200.0.0
48812026-03-10T20:08:25.578ZTRACtest_retry_backoff (dropshot_external): authn: trying Spoof local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
48822026-03-10T20:08:25.578ZDEBGtest_retry_backoff (dropshot_external): looking up silo for user local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 silo_user_id = 001de000-05e4-4000-8000-000000004007 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
48832026-03-10T20:08:25.578ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
48842026-03-10T20:08:25.578ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
48852026-03-10T20:08:25.578ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
48862026-03-10T20:08:25.580ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
48872026-03-10T20:08:25.580ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
48882026-03-10T20:08:25.581ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
48892026-03-10T20:08:25.582ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007 (silo_user), lookup_type: ById(001de000-05e4-4000-8000-000000004007) }
48902026-03-10T20:08:25.582ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
48912026-03-10T20:08:25.582ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
48922026-03-10T20:08:25.583ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
48932026-03-10T20:08:25.583ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
48942026-03-10T20:08:25.585ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48952026-03-10T20:08:25.585ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
48962026-03-10T20:08:25.585ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
48972026-03-10T20:08:25.585ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
48982026-03-10T20:08:25.587ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
48992026-03-10T20:08:25.589ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007 (silo_user), lookup_type: ById(001de000-05e4-4000-8000-000000004007) } result = Ok(())
49002026-03-10T20:08:25.589ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
49012026-03-10T20:08:25.589ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
49022026-03-10T20:08:25.589ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
49032026-03-10T20:08:25.591ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }
49042026-03-10T20:08:25.591ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
49052026-03-10T20:08:25.591ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
49062026-03-10T20:08:25.591ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
49072026-03-10T20:08:25.591ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
49082026-03-10T20:08:25.593ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49092026-03-10T20:08:25.593ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
49102026-03-10T20:08:25.593ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
49112026-03-10T20:08:25.593ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
49122026-03-10T20:08:25.598ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
49132026-03-10T20:08:25.601ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) } result = Ok(())
49142026-03-10T20:08:25.601ZTRACtest_retry_backoff (dropshot_external): authn result: Ok(Context { kind: Authenticated(Details { actor: Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }, device_token_expiration: None, credential_id: None }, Some(SiloAuthnPolicy { mapped_fleet_roles: {} })), schemes_tried: [Spoof] }) local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49152026-03-10T20:08:25.601ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49162026-03-10T20:08:25.601ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49172026-03-10T20:08:25.603ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49182026-03-10T20:08:25.613ZTRACtest_retry_backoff (dropshot_external): authorize begin action = ListChildren actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 resource = AlertReceiver { parent: Fleet, key: 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver), lookup_type: ById(2b6fbbc6-d38e-4d46-a5cd-293e75593af0) } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49192026-03-10T20:08:25.613ZTRACtest_retry_backoff (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. } authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49202026-03-10T20:08:25.613ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49212026-03-10T20:08:25.613ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49222026-03-10T20:08:25.613ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49232026-03-10T20:08:25.618ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49242026-03-10T20:08:25.622ZDEBGtest_retry_backoff (dropshot_external): authorize result action = ListChildren actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 resource = AlertReceiver { parent: Fleet, key: 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver), lookup_type: ById(2b6fbbc6-d38e-4d46-a5cd-293e75593af0) } result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49252026-03-10T20:08:25.622ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49262026-03-10T20:08:25.622ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49272026-03-10T20:08:25.623ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49282026-03-10T20:08:25.629ZINFOtest_retry_backoff (dropshot_external): request completed latency_us = 51920 local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 991ff055-15f9-4c5e-b7ce-f09235c68f40 response_code = 200 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10
49292026-03-10T20:08:25.630ZINFOtest_retry_backoff (external client test context): client received response status = 200
49302026-03-10T20:08:25.630ZINFOtest_retry_backoff (external client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:46842/v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49312026-03-10T20:08:25.632ZTRACtest_retry_backoff (dropshot_external): incoming request local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49322026-03-10T20:08:25.632ZDEBGtest_retry_backoff (dropshot_external): determined request API version local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ== version = 2026030200.0.0
49332026-03-10T20:08:25.633ZTRACtest_retry_backoff (dropshot_external): authn: trying Spoof local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49342026-03-10T20:08:25.633ZDEBGtest_retry_backoff (dropshot_external): looking up silo for user local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 silo_user_id = 001de000-05e4-4000-8000-000000004007 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49352026-03-10T20:08:25.633ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
49362026-03-10T20:08:25.633ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
49372026-03-10T20:08:25.633ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
49382026-03-10T20:08:25.635ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
49392026-03-10T20:08:25.635ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
49402026-03-10T20:08:25.636ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
49412026-03-10T20:08:25.637ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007 (silo_user), lookup_type: ById(001de000-05e4-4000-8000-000000004007) }
49422026-03-10T20:08:25.637ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
49432026-03-10T20:08:25.637ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
49442026-03-10T20:08:25.637ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
49452026-03-10T20:08:25.637ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
49462026-03-10T20:08:25.639ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49472026-03-10T20:08:25.639ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
49482026-03-10T20:08:25.639ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
49492026-03-10T20:08:25.639ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
49502026-03-10T20:08:25.640ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
49512026-03-10T20:08:25.643ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007 (silo_user), lookup_type: ById(001de000-05e4-4000-8000-000000004007) } result = Ok(())
49522026-03-10T20:08:25.643ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
49532026-03-10T20:08:25.643ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
49542026-03-10T20:08:25.643ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
49552026-03-10T20:08:25.644ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }
49562026-03-10T20:08:25.644ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
49572026-03-10T20:08:25.644ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
49582026-03-10T20:08:25.644ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
49592026-03-10T20:08:25.644ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
49602026-03-10T20:08:25.646ZTRACtest_retry_backoff (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49612026-03-10T20:08:25.646ZTRACtest_retry_backoff (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database
49622026-03-10T20:08:25.646ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {} }
49632026-03-10T20:08:25.646ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Database result = Ok(())
49642026-03-10T20:08:25.647ZDEBGtest_retry_backoff (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
49652026-03-10T20:08:25.650ZDEBGtest_retry_backoff (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003 (built_in_user), .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) } result = Ok(())
49662026-03-10T20:08:25.650ZTRACtest_retry_backoff (dropshot_external): authn result: Ok(Context { kind: Authenticated(Details { actor: Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }, device_token_expiration: None, credential_id: None }, Some(SiloAuthnPolicy { mapped_fleet_roles: {} })), schemes_tried: [Spoof] }) local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49672026-03-10T20:08:25.650ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49682026-03-10T20:08:25.650ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49692026-03-10T20:08:25.650ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49702026-03-10T20:08:25.651ZTRACtest_retry_backoff (dropshot_external): authorize begin action = ListChildren actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 resource = AlertReceiver { parent: Fleet, key: 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver), lookup_type: ById(2b6fbbc6-d38e-4d46-a5cd-293e75593af0) } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49712026-03-10T20:08:25.651ZTRACtest_retry_backoff (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. } authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49722026-03-10T20:08:25.651ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49732026-03-10T20:08:25.651ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49742026-03-10T20:08:25.651ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49752026-03-10T20:08:25.653ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49762026-03-10T20:08:25.657ZDEBGtest_retry_backoff (dropshot_external): authorize result action = ListChildren actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 resource = AlertReceiver { parent: Fleet, key: 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 (alert_receiver), lookup_type: ById(2b6fbbc6-d38e-4d46-a5cd-293e75593af0) } result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49772026-03-10T20:08:25.657ZTRACtest_retry_backoff (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 resource = Database silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49782026-03-10T20:08:25.657ZDEBGtest_retry_backoff (dropshot_external): roles authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 roles = RoleSet { roles: {} } silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49792026-03-10T20:08:25.657ZDEBGtest_retry_backoff (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007 (silo_user), silo_id: 001de000-5110-4000-8000-000000000000, .. }) authenticated = true local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 resource = Database result = Ok(()) silo_id = 001de000-5110-4000-8000-000000000000 silo_user_id = 001de000-05e4-4000-8000-000000004007 type = silo_user uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49802026-03-10T20:08:25.660ZINFOtest_retry_backoff (dropshot_external): request completed latency_us = 27437 local_addr = 127.0.0.1:46842 method = GET remote_addr = 127.0.0.1:57042 req_id = 81af5584-0c69-47fa-b3db-8ac234b4c112 response_code = 200 uri = /v1/alert-receivers/2b6fbbc6-d38e-4d46-a5cd-293e75593af0/deliveries?limit=10&page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJ0aW1lX2FuZF9pZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOlsiMjAyNi0wMy0xMFQyMDowODowOS45MDU5ODNaIiwiMTM5MDFiZjAtMDRmNi00YjhjLWEyYzEtMjc4MzI3OTI3ZmQ3Il19fQ==
49812026-03-10T20:08:25.660ZINFOtest_retry_backoff (external client test context): client received response status = 200
49822026-03-10T20:08:26.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
49832026-03-10T20:08:26.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
49842026-03-10T20:08:26.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
49852026-03-10T20:08:26.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
49862026-03-10T20:08:26.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
49872026-03-10T20:08:26.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
49882026-03-10T20:08:26.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
49892026-03-10T20:08:26.033ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
49902026-03-10T20:08:26.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
49912026-03-10T20:08:26.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
49922026-03-10T20:08:26.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
49932026-03-10T20:08:26.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
49942026-03-10T20:08:26.454ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
49952026-03-10T20:08:26.454ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
49962026-03-10T20:08:26.454ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
49972026-03-10T20:08:27.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
49982026-03-10T20:08:27.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
49992026-03-10T20:08:27.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
50002026-03-10T20:08:27.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50012026-03-10T20:08:27.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
50022026-03-10T20:08:27.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50032026-03-10T20:08:27.034ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
50042026-03-10T20:08:27.034ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50052026-03-10T20:08:27.131ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
50062026-03-10T20:08:27.442ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
50072026-03-10T20:08:27.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
50082026-03-10T20:08:27.443ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 40ac199a-2611-4f7a-a890-5ab650fe71ac uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
50092026-03-10T20:08:27.443ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 40ac199a-2611-4f7a-a890-5ab650fe71ac uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
50102026-03-10T20:08:27.443ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 40ac199a-2611-4f7a-a890-5ab650fe71ac resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
50112026-03-10T20:08:27.443ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 40ac199a-2611-4f7a-a890-5ab650fe71ac roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
50122026-03-10T20:08:27.444ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 40ac199a-2611-4f7a-a890-5ab650fe71ac resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
50132026-03-10T20:08:27.446ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3564 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 40ac199a-2611-4f7a-a890-5ab650fe71ac response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
50142026-03-10T20:08:27.447ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "40ac199a-2611-4f7a-a890-5ab650fe71ac", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:27 GMT"} })
50152026-03-10T20:08:27.447ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50162026-03-10T20:08:27.447ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c253170c-f657-41a8-ac00-90c8dd5446d8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50172026-03-10T20:08:27.447ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c253170c-f657-41a8-ac00-90c8dd5446d8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
50182026-03-10T20:08:27.447ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c253170c-f657-41a8-ac00-90c8dd5446d8 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
50192026-03-10T20:08:27.447ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c253170c-f657-41a8-ac00-90c8dd5446d8 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
50202026-03-10T20:08:27.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c253170c-f657-41a8-ac00-90c8dd5446d8 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
50212026-03-10T20:08:27.449ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2264 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c253170c-f657-41a8-ac00-90c8dd5446d8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50222026-03-10T20:08:27.449ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "c253170c-f657-41a8-ac00-90c8dd5446d8", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:27 GMT"} })
50232026-03-10T20:08:27.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
50242026-03-10T20:08:27.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50252026-03-10T20:08:27.450ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
50262026-03-10T20:08:27.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50272026-03-10T20:08:27.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50282026-03-10T20:08:27.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50292026-03-10T20:08:27.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50302026-03-10T20:08:27.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50312026-03-10T20:08:27.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50322026-03-10T20:08:28.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50332026-03-10T20:08:28.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50342026-03-10T20:08:28.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
50352026-03-10T20:08:28.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50362026-03-10T20:08:28.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
50372026-03-10T20:08:28.033ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50382026-03-10T20:08:28.034ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
50392026-03-10T20:08:28.034ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50402026-03-10T20:08:28.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
50412026-03-10T20:08:28.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50422026-03-10T20:08:28.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50432026-03-10T20:08:28.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50442026-03-10T20:08:28.454ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50452026-03-10T20:08:28.454ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50462026-03-10T20:08:28.454ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50472026-03-10T20:08:29.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50482026-03-10T20:08:29.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50492026-03-10T20:08:29.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
50502026-03-10T20:08:29.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50512026-03-10T20:08:29.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
50522026-03-10T20:08:29.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50532026-03-10T20:08:29.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
50542026-03-10T20:08:29.033ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50552026-03-10T20:08:29.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
50562026-03-10T20:08:29.443ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
50572026-03-10T20:08:29.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
50582026-03-10T20:08:29.443ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 38337dcd-c42f-4772-939e-f2952b4887ca uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
50592026-03-10T20:08:29.443ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 38337dcd-c42f-4772-939e-f2952b4887ca uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
50602026-03-10T20:08:29.444ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 38337dcd-c42f-4772-939e-f2952b4887ca resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
50612026-03-10T20:08:29.444ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 38337dcd-c42f-4772-939e-f2952b4887ca roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
50622026-03-10T20:08:29.444ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 38337dcd-c42f-4772-939e-f2952b4887ca resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
50632026-03-10T20:08:29.447ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3622 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 38337dcd-c42f-4772-939e-f2952b4887ca response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
50642026-03-10T20:08:29.447ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "38337dcd-c42f-4772-939e-f2952b4887ca", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:29 GMT"} })
50652026-03-10T20:08:29.447ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50662026-03-10T20:08:29.448ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 2e0d77c5-aca2-4525-b8e6-5bbe87f7fa62 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50672026-03-10T20:08:29.448ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 2e0d77c5-aca2-4525-b8e6-5bbe87f7fa62 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
50682026-03-10T20:08:29.448ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 2e0d77c5-aca2-4525-b8e6-5bbe87f7fa62 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
50692026-03-10T20:08:29.448ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 2e0d77c5-aca2-4525-b8e6-5bbe87f7fa62 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
50702026-03-10T20:08:29.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 2e0d77c5-aca2-4525-b8e6-5bbe87f7fa62 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
50712026-03-10T20:08:29.450ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2039 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 2e0d77c5-aca2-4525-b8e6-5bbe87f7fa62 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50722026-03-10T20:08:29.450ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e0d77c5-aca2-4525-b8e6-5bbe87f7fa62", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:29 GMT"} })
50732026-03-10T20:08:29.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
50742026-03-10T20:08:29.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50752026-03-10T20:08:29.450ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
50762026-03-10T20:08:29.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50772026-03-10T20:08:29.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50782026-03-10T20:08:29.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50792026-03-10T20:08:29.454ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50802026-03-10T20:08:29.454ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50812026-03-10T20:08:29.454ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50822026-03-10T20:08:30.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50832026-03-10T20:08:30.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50842026-03-10T20:08:30.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
50852026-03-10T20:08:30.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50862026-03-10T20:08:30.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
50872026-03-10T20:08:30.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50882026-03-10T20:08:30.034ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
50892026-03-10T20:08:30.034ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50902026-03-10T20:08:30.131ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
50912026-03-10T20:08:30.451ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50922026-03-10T20:08:30.451ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50932026-03-10T20:08:30.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50942026-03-10T20:08:30.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50952026-03-10T20:08:30.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50962026-03-10T20:08:30.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
50972026-03-10T20:08:30.562ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = reconfigurator_config_watcher iteration = 6 reason = Timeout
50982026-03-10T20:08:30.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig
50992026-03-10T20:08:30.562ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = reconfigurator_config_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51002026-03-10T20:08:30.562ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
51012026-03-10T20:08:30.562ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
51022026-03-10T20:08:30.562ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
51032026-03-10T20:08:30.565ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51042026-03-10T20:08:30.569ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig result = Ok(())
51052026-03-10T20:08:30.569ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
51062026-03-10T20:08:30.569ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
51072026-03-10T20:08:30.569ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
51082026-03-10T20:08:30.571ZDEBGtest_retry_backoff (BackgroundTasks): reconfigurator config load complete background_task = reconfigurator_config_watcher switches = Loaded(ReconfiguratorConfigView { version: 1, config: ReconfiguratorConfig { planner_enabled: false, planner_config: PlannerConfig { add_zones_with_mupdate_override: false }, tuf_repo_pruner_enabled: true }, time_modified: 2026-03-10T20:08:05.458980Z })
51092026-03-10T20:08:30.571ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = reconfigurator_config_watcher elapsed = 9.761847ms iteration = 6 status = {"config_updated":false}
51102026-03-10T20:08:31.028ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51112026-03-10T20:08:31.028ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51122026-03-10T20:08:31.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
51132026-03-10T20:08:31.029ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
51142026-03-10T20:08:31.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
51152026-03-10T20:08:31.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
51162026-03-10T20:08:31.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
51172026-03-10T20:08:31.033ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51182026-03-10T20:08:31.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
51192026-03-10T20:08:31.443ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
51202026-03-10T20:08:31.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51212026-03-10T20:08:31.443ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0faf1834-930c-4d10-9262-5ed4b852eb02 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51222026-03-10T20:08:31.443ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0faf1834-930c-4d10-9262-5ed4b852eb02 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
51232026-03-10T20:08:31.443ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0faf1834-930c-4d10-9262-5ed4b852eb02 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
51242026-03-10T20:08:31.444ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0faf1834-930c-4d10-9262-5ed4b852eb02 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
51252026-03-10T20:08:31.444ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0faf1834-930c-4d10-9262-5ed4b852eb02 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
51262026-03-10T20:08:31.447ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3823 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0faf1834-930c-4d10-9262-5ed4b852eb02 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51272026-03-10T20:08:31.447ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "0faf1834-930c-4d10-9262-5ed4b852eb02", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:31 GMT"} })
51282026-03-10T20:08:31.447ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
51292026-03-10T20:08:31.448ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = fe4f8b61-1406-4abf-98d8-fa0cb208f704 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
51302026-03-10T20:08:31.448ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = fe4f8b61-1406-4abf-98d8-fa0cb208f704 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
51312026-03-10T20:08:31.448ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = fe4f8b61-1406-4abf-98d8-fa0cb208f704 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
51322026-03-10T20:08:31.448ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = fe4f8b61-1406-4abf-98d8-fa0cb208f704 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
51332026-03-10T20:08:31.448ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = fe4f8b61-1406-4abf-98d8-fa0cb208f704 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
51342026-03-10T20:08:31.450ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2445 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = fe4f8b61-1406-4abf-98d8-fa0cb208f704 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
51352026-03-10T20:08:31.450ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe4f8b61-1406-4abf-98d8-fa0cb208f704", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:31 GMT"} })
51362026-03-10T20:08:31.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
51372026-03-10T20:08:31.450ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
51382026-03-10T20:08:31.450ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
51392026-03-10T20:08:31.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
51402026-03-10T20:08:31.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51412026-03-10T20:08:31.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
51422026-03-10T20:08:31.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51432026-03-10T20:08:31.452ZTRACtest_retry_backoff (dropshot): incoming request local_addr = 127.0.0.1:33574 method = GET remote_addr = 127.0.0.1:51522 req_id = 32aab5d9-aa92-4f8c-9552-78599495f526 uri = /913233fe-92a8-4635-9572-183f495429c4
51442026-03-10T20:08:31.458ZINFOtest_retry_backoff (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867 latency_us = 5899 local_addr = 127.0.0.1:33574 method = GET remote_addr = 127.0.0.1:51522 req_id = 32aab5d9-aa92-4f8c-9552-78599495f526 response_code = 200 uri = /913233fe-92a8-4635-9572-183f495429c4
51452026-03-10T20:08:31.458ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
51462026-03-10T20:08:31.464ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
51472026-03-10T20:08:31.464ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
51482026-03-10T20:08:31.464ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
51492026-03-10T20:08:31.464ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51502026-03-10T20:08:31.465ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51512026-03-10T20:08:31.465ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51522026-03-10T20:08:31.465ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51532026-03-10T20:08:31.465ZDEBGtest_retry_backoff (results-sink): inserting 10 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51542026-03-10T20:08:31.465ZDEBGtest_retry_backoff (results-sink-cluster): inserting 10 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51552026-03-10T20:08:31.465ZDEBGtest_retry_backoff (clickhouse-client): unrolling 10 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
51562026-03-10T20:08:31.469ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 20 sql = INSERT INTO oximeter.fields_string FORMAT Native
51572026-03-10T20:08:31.472ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 20 table_name = fields_string
51582026-03-10T20:08:31.472ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
51592026-03-10T20:08:31.473ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 10 table_name = fields_u16
51602026-03-10T20:08:31.473ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
51612026-03-10T20:08:31.475ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 10 table_name = fields_uuid
51622026-03-10T20:08:31.475ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 19 n_rows = 10 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
51632026-03-10T20:08:31.477ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 10 table_name = measurements_histogramu64
51642026-03-10T20:08:31.477ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51652026-03-10T20:08:31.566ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
51662026-03-10T20:08:32.028ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51672026-03-10T20:08:32.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51682026-03-10T20:08:32.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
51692026-03-10T20:08:32.029ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
51702026-03-10T20:08:32.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
51712026-03-10T20:08:32.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
51722026-03-10T20:08:32.033ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
51732026-03-10T20:08:32.033ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51742026-03-10T20:08:32.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
51752026-03-10T20:08:32.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51762026-03-10T20:08:32.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51772026-03-10T20:08:32.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51782026-03-10T20:08:32.453ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51792026-03-10T20:08:32.453ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51802026-03-10T20:08:32.453ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
51812026-03-10T20:08:33.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51822026-03-10T20:08:33.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51832026-03-10T20:08:33.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
51842026-03-10T20:08:33.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
51852026-03-10T20:08:33.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
51862026-03-10T20:08:33.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
51872026-03-10T20:08:33.034ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
51882026-03-10T20:08:33.034ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51892026-03-10T20:08:33.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
51902026-03-10T20:08:33.443ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
51912026-03-10T20:08:33.443ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51922026-03-10T20:08:33.444ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 00ec0ece-18d9-4705-b9f2-2a9b8762ce7a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51932026-03-10T20:08:33.444ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 00ec0ece-18d9-4705-b9f2-2a9b8762ce7a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
51942026-03-10T20:08:33.444ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 00ec0ece-18d9-4705-b9f2-2a9b8762ce7a resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
51952026-03-10T20:08:33.444ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 00ec0ece-18d9-4705-b9f2-2a9b8762ce7a roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
51962026-03-10T20:08:33.445ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 00ec0ece-18d9-4705-b9f2-2a9b8762ce7a resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
51972026-03-10T20:08:33.447ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3572 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 00ec0ece-18d9-4705-b9f2-2a9b8762ce7a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51982026-03-10T20:08:33.448ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "00ec0ece-18d9-4705-b9f2-2a9b8762ce7a", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:33 GMT"} })
51992026-03-10T20:08:33.448ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
52002026-03-10T20:08:33.448ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a935288c-b625-4931-af12-49cf91392979 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
52012026-03-10T20:08:33.448ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a935288c-b625-4931-af12-49cf91392979 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
52022026-03-10T20:08:33.448ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a935288c-b625-4931-af12-49cf91392979 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
52032026-03-10T20:08:33.448ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a935288c-b625-4931-af12-49cf91392979 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
52042026-03-10T20:08:33.449ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a935288c-b625-4931-af12-49cf91392979 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
52052026-03-10T20:08:33.450ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2412 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a935288c-b625-4931-af12-49cf91392979 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
52062026-03-10T20:08:33.451ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "a935288c-b625-4931-af12-49cf91392979", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:33 GMT"} })
52072026-03-10T20:08:33.451ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
52082026-03-10T20:08:33.451ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52092026-03-10T20:08:33.451ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
52102026-03-10T20:08:33.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52112026-03-10T20:08:33.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52122026-03-10T20:08:33.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52132026-03-10T20:08:33.454ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52142026-03-10T20:08:33.454ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52152026-03-10T20:08:33.454ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52162026-03-10T20:08:34.029ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52172026-03-10T20:08:34.029ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52182026-03-10T20:08:34.029ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
52192026-03-10T20:08:34.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
52202026-03-10T20:08:34.032ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
52212026-03-10T20:08:34.032ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
52222026-03-10T20:08:34.034ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
52232026-03-10T20:08:34.034ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52242026-03-10T20:08:34.131ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
52252026-03-10T20:08:34.452ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52262026-03-10T20:08:34.452ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52272026-03-10T20:08:34.453ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52282026-03-10T20:08:34.454ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52292026-03-10T20:08:34.454ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52302026-03-10T20:08:34.454ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52312026-03-10T20:08:35.027ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52322026-03-10T20:08:35.027ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52332026-03-10T20:08:35.027ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
52342026-03-10T20:08:35.028ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
52352026-03-10T20:08:35.030ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
52362026-03-10T20:08:35.030ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
52372026-03-10T20:08:35.031ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
52382026-03-10T20:08:35.031ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52392026-03-10T20:08:35.122ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
52402026-03-10T20:08:35.416ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
52412026-03-10T20:08:35.416ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52422026-03-10T20:08:35.416ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 69c0f3da-6107-44f7-86cd-73713667077b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52432026-03-10T20:08:35.416ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 69c0f3da-6107-44f7-86cd-73713667077b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
52442026-03-10T20:08:35.416ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 69c0f3da-6107-44f7-86cd-73713667077b resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
52452026-03-10T20:08:35.416ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 69c0f3da-6107-44f7-86cd-73713667077b roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
52462026-03-10T20:08:35.417ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 69c0f3da-6107-44f7-86cd-73713667077b resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
52472026-03-10T20:08:35.420ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3799 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 69c0f3da-6107-44f7-86cd-73713667077b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52482026-03-10T20:08:35.420ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "69c0f3da-6107-44f7-86cd-73713667077b", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
52492026-03-10T20:08:35.420ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
52502026-03-10T20:08:35.420ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c8db7923-a7c0-453b-9466-19739df35168 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
52512026-03-10T20:08:35.420ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c8db7923-a7c0-453b-9466-19739df35168 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
52522026-03-10T20:08:35.420ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c8db7923-a7c0-453b-9466-19739df35168 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
52532026-03-10T20:08:35.420ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c8db7923-a7c0-453b-9466-19739df35168 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
52542026-03-10T20:08:35.421ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c8db7923-a7c0-453b-9466-19739df35168 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
52552026-03-10T20:08:35.423ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2442 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = c8db7923-a7c0-453b-9466-19739df35168 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
52562026-03-10T20:08:35.423ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8db7923-a7c0-453b-9466-19739df35168", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
52572026-03-10T20:08:35.423ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
52582026-03-10T20:08:35.423ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52592026-03-10T20:08:35.423ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
52602026-03-10T20:08:35.424ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52612026-03-10T20:08:35.424ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52622026-03-10T20:08:35.425ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52632026-03-10T20:08:35.425ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52642026-03-10T20:08:35.425ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52652026-03-10T20:08:35.425ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
52662026-03-10T20:08:35.524ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = nat_garbage_collector iteration = 2 reason = Timeout
52672026-03-10T20:08:35.524ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = nat_garbage_collector resource = Database
52682026-03-10T20:08:35.524ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = nat_garbage_collector roles = RoleSet { roles: {} }
52692026-03-10T20:08:35.525ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = nat_garbage_collector resource = Database result = Ok(())
52702026-03-10T20:08:35.525ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
52712026-03-10T20:08:35.525ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = bfd_manager resource = Database
52722026-03-10T20:08:35.525ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
52732026-03-10T20:08:35.526ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = bfd_manager resource = Database result = Ok(())
52742026-03-10T20:08:35.526ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
52752026-03-10T20:08:35.526ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = inventory_loader iteration = 4 reason = Timeout
52762026-03-10T20:08:35.526ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Inventory
52772026-03-10T20:08:35.526ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = inventory_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52782026-03-10T20:08:35.526ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database
52792026-03-10T20:08:35.526ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {} }
52802026-03-10T20:08:35.527ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database result = Ok(())
52812026-03-10T20:08:35.527ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = reconfigurator_config_watcher iteration = 7 reason = Timeout
52822026-03-10T20:08:35.527ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig
52832026-03-10T20:08:35.527ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = reconfigurator_config_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52842026-03-10T20:08:35.527ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
52852026-03-10T20:08:35.527ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
52862026-03-10T20:08:35.527ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
52872026-03-10T20:08:35.529ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
52882026-03-10T20:08:35.529ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
52892026-03-10T20:08:35.529ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 25.917\u{b5}s iteration = 3 status = {"error":"task disabled"}
52902026-03-10T20:08:35.529ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
52912026-03-10T20:08:35.529ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database
52922026-03-10T20:08:35.529ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52932026-03-10T20:08:35.529ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52942026-03-10T20:08:35.529ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
52952026-03-10T20:08:35.529ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Fleet
52962026-03-10T20:08:35.529ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52972026-03-10T20:08:35.529ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database
52982026-03-10T20:08:35.529ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52992026-03-10T20:08:35.529ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53002026-03-10T20:08:35.530ZTRACtest_retry_backoff (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
53012026-03-10T20:08:35.530ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53022026-03-10T20:08:35.534ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Inventory result = Ok(())
53032026-03-10T20:08:35.534ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database
53042026-03-10T20:08:35.534ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = inventory_loader roles = RoleSet { roles: {} }
53052026-03-10T20:08:35.534ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = inventory_loader resource = Database result = Ok(())
53062026-03-10T20:08:35.535ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 9.477063ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
53072026-03-10T20:08:35.535ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53082026-03-10T20:08:35.539ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig result = Ok(())
53092026-03-10T20:08:35.540ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
53102026-03-10T20:08:35.540ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
53112026-03-10T20:08:35.540ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
53122026-03-10T20:08:35.541ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53132026-03-10T20:08:35.544ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
53142026-03-10T20:08:35.544ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database
53152026-03-10T20:08:35.544ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53162026-03-10T20:08:35.544ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53172026-03-10T20:08:35.545ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229 (sled), lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
53182026-03-10T20:08:35.545ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53192026-03-10T20:08:35.545ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database
53202026-03-10T20:08:35.545ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53212026-03-10T20:08:35.545ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53222026-03-10T20:08:35.546ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
53232026-03-10T20:08:35.546ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database
53242026-03-10T20:08:35.546ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53252026-03-10T20:08:35.546ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database result = Ok(())
53262026-03-10T20:08:35.546ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
53272026-03-10T20:08:35.546ZINFOtest_retry_backoff (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
53282026-03-10T20:08:35.546ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = instance_updater elapsed = 62.326\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
53292026-03-10T20:08:35.546ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
53302026-03-10T20:08:35.546ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Fleet
53312026-03-10T20:08:35.546ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53322026-03-10T20:08:35.546ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Database
53332026-03-10T20:08:35.546ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
53342026-03-10T20:08:35.547ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Database result = Ok(())
53352026-03-10T20:08:35.547ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = vpc_route_manager iteration = 3 reason = Timeout
53362026-03-10T20:08:35.547ZINFOtest_retry_backoff (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
53372026-03-10T20:08:35.547ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet
53382026-03-10T20:08:35.547ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53392026-03-10T20:08:35.547ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
53402026-03-10T20:08:35.547ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53412026-03-10T20:08:35.547ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53422026-03-10T20:08:35.547ZTRACtest_retry_backoff (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: a5 10 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49987 req_id = beda655f-435a-4aca-bed4-2559e6772279
53432026-03-10T20:08:35.548ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42256,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:49987 req_id = beda655f-435a-4aca-bed4-2559e6772279
53442026-03-10T20:08:35.548ZDEBGtest_retry_backoff (store): query key key = _dendrite._tcp
53452026-03-10T20:08:35.548ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53462026-03-10T20:08:35.548ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:49987 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = beda655f-435a-4aca-bed4-2559e6772279
53472026-03-10T20:08:35.548ZDEBGtest_retry_backoff (BackgroundTasks): latest inventory collection is unchanged background_task = inventory_loader original_id = 217d854a-1530-402d-b94f-a3562c53f830 original_time_started = 2026-03-10 20:08:08.140360 UTC
53482026-03-10T20:08:35.548ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = inventory_loader elapsed = 23.779651ms iteration = 4 status = {"Loaded":{"collection_id":"217d854a-1530-402d-b94f-a3562c53f830","time_loaded":"2026-03-10T20:08:35.526685912Z","time_started":"2026-03-10T20:08:08.140360Z"}}
53492026-03-10T20:08:35.549ZDEBGtest_retry_backoff (BackgroundTasks): reconfigurator config load complete background_task = reconfigurator_config_watcher switches = Loaded(ReconfiguratorConfigView { version: 1, config: ReconfiguratorConfig { planner_enabled: false, planner_config: PlannerConfig { add_zones_with_mupdate_override: false }, tuf_repo_pruner_enabled: true }, time_modified: 2026-03-10T20:08:05.458980Z })
53502026-03-10T20:08:35.549ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = reconfigurator_config_watcher elapsed = 23.708123ms iteration = 7 status = {"config_updated":false}
53512026-03-10T20:08:35.550ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53522026-03-10T20:08:35.554ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229 (sled), lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
53532026-03-10T20:08:35.554ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database
53542026-03-10T20:08:35.554ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53552026-03-10T20:08:35.554ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53562026-03-10T20:08:35.555ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53572026-03-10T20:08:35.558ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
53582026-03-10T20:08:35.558ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet
53592026-03-10T20:08:35.558ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53602026-03-10T20:08:35.558ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
53612026-03-10T20:08:35.558ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53622026-03-10T20:08:35.558ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53632026-03-10T20:08:35.559ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53642026-03-10T20:08:35.561ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Fleet result = Ok(())
53652026-03-10T20:08:35.561ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Database
53662026-03-10T20:08:35.562ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
53672026-03-10T20:08:35.562ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = instance_watcher resource = Database result = Ok(())
53682026-03-10T20:08:35.563ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6335, tv_nsec: 460442189 } })
53692026-03-10T20:08:35.563ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
53702026-03-10T20:08:35.563ZTRACtest_retry_backoff (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 59 e6 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 Y............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62538 req_id = 3aba4abc-442e-4dae-a1fa-52073eb58ddd
53712026-03-10T20:08:35.563ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23014,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:62538 req_id = 3aba4abc-442e-4dae-a1fa-52073eb58ddd
53722026-03-10T20:08:35.563ZDEBGtest_retry_backoff (store): query key key = _dendrite._tcp
53732026-03-10T20:08:35.563ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53742026-03-10T20:08:35.563ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:62538 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = 3aba4abc-442e-4dae-a1fa-52073eb58ddd
53752026-03-10T20:08:35.563ZTRACtest_retry_backoff (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
53762026-03-10T20:08:35.564ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database
53772026-03-10T20:08:35.564ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53782026-03-10T20:08:35.564ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database result = Ok(())
53792026-03-10T20:08:35.566ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database
53802026-03-10T20:08:35.566ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53812026-03-10T20:08:35.566ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53822026-03-10T20:08:35.566ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53832026-03-10T20:08:35.571ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
53842026-03-10T20:08:35.571ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
53852026-03-10T20:08:35.571ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53862026-03-10T20:08:35.571ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53872026-03-10T20:08:35.571ZDEBGtest_retry_backoff (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6335, tv_nsec: 468882605 } })
53882026-03-10T20:08:35.571ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
53892026-03-10T20:08:35.571ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 2 reason = Timeout
53902026-03-10T20:08:35.571ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
53912026-03-10T20:08:35.571ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = fm_sitrep_loader iteration = 3 reason = Timeout
53922026-03-10T20:08:35.571ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Fleet
53932026-03-10T20:08:35.571ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = fm_sitrep_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53942026-03-10T20:08:35.571ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database
53952026-03-10T20:08:35.571ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_loader roles = RoleSet { roles: {} }
53962026-03-10T20:08:35.573ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database result = Ok(())
53972026-03-10T20:08:35.573ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = fm_sitrep_gc iteration = 3 reason = Timeout
53982026-03-10T20:08:35.573ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
53992026-03-10T20:08:35.573ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
54002026-03-10T20:08:35.573ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
54012026-03-10T20:08:35.573ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database
54022026-03-10T20:08:35.573ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
54032026-03-10T20:08:35.574ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
54042026-03-10T20:08:35.574ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet
54052026-03-10T20:08:35.574ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = fm_sitrep_gc resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54062026-03-10T20:08:35.574ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
54072026-03-10T20:08:35.574ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
54082026-03-10T20:08:35.575ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
54092026-03-10T20:08:35.615ZINFOtest_retry_backoff (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
54102026-03-10T20:08:35.615ZDEBGtest_retry_backoff (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:45969/vpc-routes
54112026-03-10T20:08:35.616ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database
54122026-03-10T20:08:35.616ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
54132026-03-10T20:08:35.616ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
54142026-03-10T20:08:35.617ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54152026-03-10T20:08:35.621ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Fleet result = Ok(())
54162026-03-10T20:08:35.621ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database
54172026-03-10T20:08:35.621ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_loader roles = RoleSet { roles: {} }
54182026-03-10T20:08:35.621ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_loader resource = Database result = Ok(())
54192026-03-10T20:08:35.622ZINFOtest_retry_backoff (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
54202026-03-10T20:08:35.623ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 81.750777ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
54212026-03-10T20:08:35.623ZINFOtest_retry_backoff (dropshot): accepted connection local_addr = [::1]:45969 remote_addr = [::1]:61786 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
54222026-03-10T20:08:35.624ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7a 73 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e zs..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:23444 req_id = 2ac09ad1-5bb8-4b61-8759-43e4cbce871b
54232026-03-10T20:08:35.624ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31347,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:23444 req_id = 2ac09ad1-5bb8-4b61-8759-43e4cbce871b
54242026-03-10T20:08:35.624ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54252026-03-10T20:08:35.624ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:23444 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 2ac09ad1-5bb8-4b61-8759-43e4cbce871b
54262026-03-10T20:08:35.624ZTRACtest_retry_backoff (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 19 a1 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:44349 req_id = f620e6d8-de36-42f1-94b9-04cfe5188a50
54272026-03-10T20:08:35.624ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6561,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:44349 req_id = f620e6d8-de36-42f1-94b9-04cfe5188a50
54282026-03-10T20:08:35.625ZDEBGtest_retry_backoff (store): query key key = _dendrite._tcp
54292026-03-10T20:08:35.625ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54302026-03-10T20:08:35.625ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:44349 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = f620e6d8-de36-42f1-94b9-04cfe5188a50
54312026-03-10T20:08:35.625ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54322026-03-10T20:08:35.629ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet result = Ok(())
54332026-03-10T20:08:35.631ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = fm_sitrep_loader elapsed = 64.11304ms iteration = 3 status = "NoSitrep"
54342026-03-10T20:08:35.631ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
54352026-03-10T20:08:35.631ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
54362026-03-10T20:08:35.632ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
54372026-03-10T20:08:35.669ZTRACtest_retry_backoff (DpdClient): client request background_task = nat_garbage_collector body = None method = GET uri = http://[::1]:41467/switch/identifiers
54382026-03-10T20:08:35.670ZDEBGtest_retry_backoff (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6335, tv_nsec: 574917233 } })
54392026-03-10T20:08:35.670ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
54402026-03-10T20:08:35.670ZTRACtest_retry_backoff (dropshot): incoming request local_addr = [::1]:45969 method = GET remote_addr = [::1]:61786 req_id = a3253b3e-3c06-4f0e-a4f1-0356c5dd7c5a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
54412026-03-10T20:08:35.670ZDEBGtest_retry_backoff (dropshot): determined request API version local_addr = [::1]:45969 method = GET remote_addr = [::1]:61786 req_id = a3253b3e-3c06-4f0e-a4f1-0356c5dd7c5a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes version = 27.0.0
54422026-03-10T20:08:35.670ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 54 d1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e T...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56981 req_id = 6a9032ed-f2c1-4b55-a910-521f2f631fd4
54432026-03-10T20:08:35.670ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21713,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:56981 req_id = 6a9032ed-f2c1-4b55-a910-521f2f631fd4
54442026-03-10T20:08:35.671ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54452026-03-10T20:08:35.671ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56981 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 6a9032ed-f2c1-4b55-a910-521f2f631fd4
54462026-03-10T20:08:35.671ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 62 52 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 bR..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:10836 req_id = e79994cb-b09a-4cf7-8abf-2da7630081cd
54472026-03-10T20:08:35.671ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25170,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:10836 req_id = e79994cb-b09a-4cf7-8abf-2da7630081cd
54482026-03-10T20:08:35.671ZDEBGtest_retry_backoff (store): query key key = _mgs._tcp
54492026-03-10T20:08:35.671ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54502026-03-10T20:08:35.671ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:10836 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 43524, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = e79994cb-b09a-4cf7-8abf-2da7630081cd
54512026-03-10T20:08:35.671ZINFOtest_retry_backoff (dropshot): request completed latency_us = 988 local_addr = [::1]:45969 method = GET remote_addr = [::1]:61786 req_id = a3253b3e-3c06-4f0e-a4f1-0356c5dd7c5a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
54522026-03-10T20:08:35.671ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
54532026-03-10T20:08:35.672ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet
54542026-03-10T20:08:35.672ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = fm_sitrep_gc resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54552026-03-10T20:08:35.672ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database
54562026-03-10T20:08:35.672ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {} }
54572026-03-10T20:08:35.672ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Database result = Ok(())
54582026-03-10T20:08:35.673ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Fleet
54592026-03-10T20:08:35.673ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54602026-03-10T20:08:35.674ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database
54612026-03-10T20:08:35.674ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54622026-03-10T20:08:35.674ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database result = Ok(())
54632026-03-10T20:08:35.675ZINFOtest_retry_backoff (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
54642026-03-10T20:08:35.675ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
54652026-03-10T20:08:35.675ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 43524, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6335, tv_nsec: 580791528 } })
54662026-03-10T20:08:35.675ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
54672026-03-10T20:08:35.676ZDEBGtest_retry_backoff (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:45969/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3253b3e-3c06-4f0e-a4f1-0356c5dd7c5a", "content-length": "2", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
54682026-03-10T20:08:35.676ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database
54692026-03-10T20:08:35.676ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
54702026-03-10T20:08:35.676ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = vpc_route_manager resource = Database result = Ok(())
54712026-03-10T20:08:35.677ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54722026-03-10T20:08:35.680ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Fleet result = Ok(())
54732026-03-10T20:08:35.681ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Fleet
54742026-03-10T20:08:35.681ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54752026-03-10T20:08:35.681ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database
54762026-03-10T20:08:35.681ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54772026-03-10T20:08:35.681ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database result = Ok(())
54782026-03-10T20:08:35.682ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = fm_sitrep_gc roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54792026-03-10T20:08:35.691ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = fm_sitrep_gc resource = Fleet result = Ok(())
54802026-03-10T20:08:35.695ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 27 78 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 'x..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:12562 req_id = 72a9beaa-73d4-47a6-98dc-ae33ca2c7c34
54812026-03-10T20:08:35.695ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10104,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:12562 req_id = 72a9beaa-73d4-47a6-98dc-ae33ca2c7c34
54822026-03-10T20:08:35.695ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54832026-03-10T20:08:35.695ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:12562 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 72a9beaa-73d4-47a6-98dc-ae33ca2c7c34
54842026-03-10T20:08:35.695ZTRACtest_retry_backoff (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 4c 16 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 L............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56857 req_id = c9170a10-db98-4d29-9e45-c4597844a105
54852026-03-10T20:08:35.695ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19478,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:56857 req_id = c9170a10-db98-4d29-9e45-c4597844a105
54862026-03-10T20:08:35.695ZDEBGtest_retry_backoff (store): query key key = _dendrite._tcp
54872026-03-10T20:08:35.695ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54882026-03-10T20:08:35.695ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:56857 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = c9170a10-db98-4d29-9e45-c4597844a105
54892026-03-10T20:08:35.696ZTRACtest_retry_backoff (DpdClient): client response background_task = nat_garbage_collector result = Ok(Response { url: "http://[::1]:41467/switch/identifiers", status: 200, headers: {"content-type": "application/json", "x-request-id": "78a2b19c-3e84-4b78-8325-3fdcced2cc8c", "content-length": "717", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
54902026-03-10T20:08:35.696ZERROtest_retry_backoff (BackgroundTasks): dendrite for switch1 is unavailable, cannot perform nat cleanup background_task = nat_garbage_collector
54912026-03-10T20:08:35.696ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = nat_garbage_collector elapsed = 183.331928ms iteration = 2 status = {"error":"dendrite for switch1 is unavailable, cannot perform nat cleanup"}
54922026-03-10T20:08:35.696ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54932026-03-10T20:08:35.700ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Fleet result = Ok(())
54942026-03-10T20:08:35.700ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database
54952026-03-10T20:08:35.700ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
54962026-03-10T20:08:35.700ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = v2p_manager resource = Database result = Ok(())
54972026-03-10T20:08:35.702ZINFOtest_retry_backoff (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
54982026-03-10T20:08:35.703ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
54992026-03-10T20:08:35.703ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6335, tv_nsec: 610168722 } })
55002026-03-10T20:08:35.703ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
55012026-03-10T20:08:35.703ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 14 c0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54757 req_id = b0902923-59b5-47e4-a521-c1a9614828e8
55022026-03-10T20:08:35.703ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5312,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:54757 req_id = b0902923-59b5-47e4-a521-c1a9614828e8
55032026-03-10T20:08:35.703ZDEBGtest_retry_backoff (store): query key key = _mgs._tcp
55042026-03-10T20:08:35.703ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55052026-03-10T20:08:35.703ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:54757 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 43524, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = b0902923-59b5-47e4-a521-c1a9614828e8
55062026-03-10T20:08:35.703ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e0 a7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63478 req_id = 725e77c9-59bc-4e52-8e87-89be3d3acf60
55072026-03-10T20:08:35.703ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57511,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:63478 req_id = 725e77c9-59bc-4e52-8e87-89be3d3acf60
55082026-03-10T20:08:35.704ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55092026-03-10T20:08:35.704ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63478 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 725e77c9-59bc-4e52-8e87-89be3d3acf60
55102026-03-10T20:08:35.709ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 43524, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6335, tv_nsec: 616680611 } })
55112026-03-10T20:08:35.709ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
55122026-03-10T20:08:35.754ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[::1]:43524 method = GET uri = http://[::1]:43524/local/all-sp-ids
55132026-03-10T20:08:35.758ZINFOtest_retry_backoff (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = SimGimlet00
55142026-03-10T20:08:35.758ZDEBGtest_retry_backoff (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:45969/eip-gateways
55152026-03-10T20:08:35.758ZINFOtest_retry_backoff (BackgroundTasks): deleted 0 of 0 sitreps sitreps background_task = fm_sitrep_gc case_ereports_deleted = 0 cases_deleted = 0 ids = [] sitreps_deleted = 0
55162026-03-10T20:08:35.758ZTRACtest_retry_backoff (BackgroundTasks): all 0 orphaned sitreps in this batch have already been deleted background_task = fm_sitrep_gc
55172026-03-10T20:08:35.758ZINFOtest_retry_backoff (BackgroundTasks): sitrep garbage collection found 0 orphaned sitreps, deleted 0 background_task = fm_sitrep_gc
55182026-03-10T20:08:35.758ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = fm_sitrep_gc elapsed = 198.851739ms iteration = 3 status = {"errors":[],"orphaned_sitreps_deleted":0,"orphaned_sitreps_found":0}
55192026-03-10T20:08:35.759ZTRACtest_retry_backoff (dropshot): incoming request local_addr = [::1]:45969 method = PUT remote_addr = [::1]:61786 req_id = be4a7f73-d12a-4427-bd6d-aeefe3b11dbb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
55202026-03-10T20:08:35.759ZDEBGtest_retry_backoff (dropshot): determined request API version local_addr = [::1]:45969 method = PUT remote_addr = [::1]:61786 req_id = be4a7f73-d12a-4427-bd6d-aeefe3b11dbb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways version = 27.0.0
55212026-03-10T20:08:35.759ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 09 57 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .W..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:36583 req_id = dd5cf7f2-deb0-44f0-ab63-90c10cfe1ccc
55222026-03-10T20:08:35.759ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2391,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:36583 req_id = dd5cf7f2-deb0-44f0-ab63-90c10cfe1ccc
55232026-03-10T20:08:35.760ZDEBGtest_retry_backoff (store): query key key = _mgs._tcp
55242026-03-10T20:08:35.760ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55252026-03-10T20:08:35.760ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:36583 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 43524, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = dd5cf7f2-deb0-44f0-ab63-90c10cfe1ccc
55262026-03-10T20:08:35.760ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d1 e1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:10901 req_id = d4cdccd9-d832-43f2-9ca6-cf2b483d88e9
55272026-03-10T20:08:35.760ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53729,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:10901 req_id = d4cdccd9-d832-43f2-9ca6-cf2b483d88e9
55282026-03-10T20:08:35.760ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55292026-03-10T20:08:35.760ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:10901 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = d4cdccd9-d832-43f2-9ca6-cf2b483d88e9
55302026-03-10T20:08:35.762ZINFOtest_retry_backoff (dropshot): request completed latency_us = 3057 local_addr = [::1]:45969 method = PUT remote_addr = [::1]:61786 req_id = be4a7f73-d12a-4427-bd6d-aeefe3b11dbb response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
55312026-03-10T20:08:35.763ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 43524, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6335, tv_nsec: 673818503 } })
55322026-03-10T20:08:35.763ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
55332026-03-10T20:08:35.797ZTRACtest_retry_backoff (DpdClient): client request background_task = service_zone_nat_tracker body = None method = GET uri = http://[::1]:41467/switch/identifiers
55342026-03-10T20:08:35.798ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4c bc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e L...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:47660 req_id = 70cbda11-4d88-43eb-ad31-8e657dc7b9a3
55352026-03-10T20:08:35.798ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19644,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:47660 req_id = 70cbda11-4d88-43eb-ad31-8e657dc7b9a3
55362026-03-10T20:08:35.798ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55372026-03-10T20:08:35.798ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:47660 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 70cbda11-4d88-43eb-ad31-8e657dc7b9a3
55382026-03-10T20:08:35.798ZDEBGtest_retry_backoff (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:45969/eip-gateways", status: 204, headers: {"x-request-id": "be4a7f73-d12a-4427-bd6d-aeefe3b11dbb", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
55392026-03-10T20:08:35.798ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 268.122758ms iteration = 3 status = {}
55402026-03-10T20:08:35.832ZDEBGtest_retry_backoff (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:45969/v2p
55412026-03-10T20:08:35.833ZINFOtest_retry_backoff (dropshot): accepted connection local_addr = [::1]:45969 remote_addr = [::1]:47691 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
55422026-03-10T20:08:35.867ZINFOtest_retry_backoff (BackgroundTasks): determining switch slot managed by switch zone background_task = bfd_manager zone_address = ::1
55432026-03-10T20:08:35.867ZDEBGtest_retry_backoff (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:43524/local/switch-id
55442026-03-10T20:08:35.869ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f76b6fd-f1d6-41b1-bcbb-11b156f9deef", "content-length": "105", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
55452026-03-10T20:08:35.869ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet slot = 1 sp_type = switch
55462026-03-10T20:08:35.870ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = sp_ereport_ingester resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet slot = 1 sp_type = switch
55472026-03-10T20:08:35.870ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 1 sp_type = switch
55482026-03-10T20:08:35.870ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = switch
55492026-03-10T20:08:35.870ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = switch
55502026-03-10T20:08:35.871ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet slot = 0 sp_type = sled
55512026-03-10T20:08:35.871ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = sp_ereport_ingester resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet slot = 0 sp_type = sled
55522026-03-10T20:08:35.871ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 0 sp_type = sled
55532026-03-10T20:08:35.871ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
55542026-03-10T20:08:35.871ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
55552026-03-10T20:08:35.872ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet slot = 0 sp_type = switch
55562026-03-10T20:08:35.872ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = sp_ereport_ingester resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet slot = 0 sp_type = switch
55572026-03-10T20:08:35.872ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 0 sp_type = switch
55582026-03-10T20:08:35.872ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
55592026-03-10T20:08:35.872ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
55602026-03-10T20:08:35.873ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet slot = 1 sp_type = sled
55612026-03-10T20:08:35.873ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = sp_ereport_ingester resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet slot = 1 sp_type = sled
55622026-03-10T20:08:35.873ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 1 sp_type = sled
55632026-03-10T20:08:35.873ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
55642026-03-10T20:08:35.873ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
55652026-03-10T20:08:35.874ZTRACtest_retry_backoff (dropshot): incoming request local_addr = [::1]:45969 method = GET remote_addr = [::1]:47691 req_id = 6c42e51d-2412-4484-99b4-ad1360546f06 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
55662026-03-10T20:08:35.874ZDEBGtest_retry_backoff (dropshot): determined request API version local_addr = [::1]:45969 method = GET remote_addr = [::1]:47691 req_id = 6c42e51d-2412-4484-99b4-ad1360546f06 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p version = 27.0.0
55672026-03-10T20:08:35.874ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fd 2c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .,..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:41074 req_id = 3bbc5389-2a77-40c3-b4b7-d20b2c7f0080
55682026-03-10T20:08:35.874ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64812,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:41074 req_id = 3bbc5389-2a77-40c3-b4b7-d20b2c7f0080
55692026-03-10T20:08:35.875ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55702026-03-10T20:08:35.875ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:41074 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 3bbc5389-2a77-40c3-b4b7-d20b2c7f0080
55712026-03-10T20:08:35.875ZTRACtest_retry_backoff (DpdClient): client response background_task = service_zone_nat_tracker result = Ok(Response { url: "http://[::1]:41467/switch/identifiers", status: 200, headers: {"content-type": "application/json", "x-request-id": "766968b4-8478-4b44-936f-53e8b7a6ddf1", "content-length": "717", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
55722026-03-10T20:08:35.875ZTRACtest_retry_backoff (DpdClient): client request background_task = service_zone_nat_tracker body = None method = POST uri = http://[::1]:41467/rpw/nat/trigger
55732026-03-10T20:08:35.875ZINFOtest_retry_backoff (dropshot): request completed latency_us = 1062 local_addr = [::1]:45969 method = GET remote_addr = [::1]:47691 req_id = 6c42e51d-2412-4484-99b4-ad1360546f06 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
55742026-03-10T20:08:35.910ZINFOtest_retry_backoff (BackgroundTasks): determining switch slot managed by switch zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
55752026-03-10T20:08:35.910ZDEBGtest_retry_backoff (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:43524/local/switch-id
55762026-03-10T20:08:35.911ZTRACtest_retry_backoff (DpdClient): client response background_task = service_zone_nat_tracker result = Ok(Response { url: "http://[::1]:41467/rpw/nat/trigger", status: 200, headers: {"content-type": "application/json", "x-request-id": "89b48c7d-862b-4e4b-a67b-73a98cc8afe8", "content-length": "4", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
55772026-03-10T20:08:35.911ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 408.49716ms iteration = 2 status = 3
55782026-03-10T20:08:35.912ZDEBGtest_retry_backoff (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:45969/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c42e51d-2412-4484-99b4-ad1360546f06", "content-length": "2", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
55792026-03-10T20:08:35.912ZINFOtest_retry_backoff (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = SimGimlet00
55802026-03-10T20:08:35.912ZINFOtest_retry_backoff (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = SimGimlet00
55812026-03-10T20:08:35.912ZINFOtest_retry_backoff (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = SimGimlet00
55822026-03-10T20:08:35.912ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 390.453819ms iteration = 2 status = {}
55832026-03-10T20:08:35.912ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
55842026-03-10T20:08:35.916ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
55852026-03-10T20:08:35.916ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 0 sp_type = sled
55862026-03-10T20:08:35.916ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
55872026-03-10T20:08:35.916ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
55882026-03-10T20:08:35.917ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = switch
55892026-03-10T20:08:35.920ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = switch
55902026-03-10T20:08:35.920ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 1 sp_type = switch
55912026-03-10T20:08:35.920ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = switch
55922026-03-10T20:08:35.920ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = switch
55932026-03-10T20:08:35.921ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
55942026-03-10T20:08:35.924ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
55952026-03-10T20:08:35.924ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 1 sp_type = sled
55962026-03-10T20:08:35.924ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
55972026-03-10T20:08:35.924ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
55982026-03-10T20:08:35.925ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
55992026-03-10T20:08:35.927ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
56002026-03-10T20:08:35.927ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 0 sp_type = switch
56012026-03-10T20:08:35.928ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
56022026-03-10T20:08:35.928ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
56032026-03-10T20:08:35.930ZDEBGtest_retry_backoff (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[::1]:43524/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd8c3786-688c-4363-9f7e-79f050bb2ac5", "content-length": "26", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
56042026-03-10T20:08:35.930ZINFOtest_retry_backoff (BackgroundTasks): identified switch slot for switch zone background_task = bfd_manager slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = ::1
56052026-03-10T20:08:35.930ZINFOtest_retry_backoff (BackgroundTasks): completed mapping switch zones to switch slots background_task = bfd_manager mappings = {\n Switch0: ::1,\n}
56062026-03-10T20:08:35.930ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgd._tcp.control-plane.oxide.internal
56072026-03-10T20:08:35.934ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 14 4a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .J..........._mg\n0010: 64 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 d._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:12158 req_id = 928cd36b-e83d-40e1-82e6-a144c04bf73a
56082026-03-10T20:08:35.934ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5194,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgd._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgd._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 100,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:12158 req_id = 928cd36b-e83d-40e1-82e6-a144c04bf73a
56092026-03-10T20:08:35.934ZDEBGtest_retry_backoff (store): query key key = _mgd._tcp
56102026-03-10T20:08:35.935ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56112026-03-10T20:08:35.935ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:12158 query = LowerQuery { name: LowerName(Name("_mgd._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgd._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgd._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 52170, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = 928cd36b-e83d-40e1-82e6-a144c04bf73a
56122026-03-10T20:08:35.935ZDEBGtest_retry_backoff (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Ok(Response { url: "http://[::1]:43524/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0348cfa-c39d-4025-8995-a8b22a553839", "content-length": "26", "date": "Tue, 10 Mar 2026 20:08:35 GMT"} })
56132026-03-10T20:08:35.935ZINFOtest_retry_backoff (BackgroundTasks): identified switch slot for switch zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = ::1
56142026-03-10T20:08:35.935ZINFOtest_retry_backoff (BackgroundTasks): completed mapping switch zones to switch slots background_task = switch_port_config_manager mappings = {\n Switch0: ::1,\n} rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
56152026-03-10T20:08:36.010ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
56162026-03-10T20:08:36.010ZDEBGtest_retry_backoff (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [::1]:43524 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
56172026-03-10T20:08:36.010ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[::1]:43524 method = POST uri = http://[::1]:43524/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
56182026-03-10T20:08:36.011ZDEBGtest_retry_backoff (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [::1]:43524 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
56192026-03-10T20:08:36.011ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[::1]:43524 method = POST uri = http://[::1]:43524/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
56202026-03-10T20:08:36.011ZDEBGtest_retry_backoff (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [::1]:43524 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
56212026-03-10T20:08:36.011ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[::1]:43524 method = POST uri = http://[::1]:43524/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
56222026-03-10T20:08:36.011ZDEBGtest_retry_backoff (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [::1]:43524 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = switch start_ena = None
56232026-03-10T20:08:36.011ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[::1]:43524 method = POST uri = http://[::1]:43524/sp/switch/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
56242026-03-10T20:08:36.012ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgd._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgd._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgd._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 52170, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6335, tv_nsec: 939981646 } })
56252026-03-10T20:08:36.012ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
56262026-03-10T20:08:36.012ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
56272026-03-10T20:08:36.012ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
56282026-03-10T20:08:36.015ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
56292026-03-10T20:08:36.016ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
56302026-03-10T20:08:36.016ZTRACtest_retry_backoff (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ea 50 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .P..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:31745 req_id = 69a29d9f-da8a-4f40-ad11-e584e8baa452
56312026-03-10T20:08:36.017ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59984,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:31745 req_id = 69a29d9f-da8a-4f40-ad11-e584e8baa452
56322026-03-10T20:08:36.017ZDEBGtest_retry_backoff (store): query key key = _dendrite._tcp
56332026-03-10T20:08:36.017ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56342026-03-10T20:08:36.017ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:31745 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = 69a29d9f-da8a-4f40-ad11-e584e8baa452
56352026-03-10T20:08:36.018ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 41467, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6335, tv_nsec: 946010040 } })
56362026-03-10T20:08:36.018ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
56372026-03-10T20:08:36.018ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: db 84 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:7192 req_id = 54813d56-8d33-4f10-9f45-9677d67f60f1
56382026-03-10T20:08:36.018ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56196,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:7192 req_id = 54813d56-8d33-4f10-9f45-9677d67f60f1
56392026-03-10T20:08:36.018ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56402026-03-10T20:08:36.018ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:7192 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 54813d56-8d33-4f10-9f45-9677d67f60f1
56412026-03-10T20:08:36.020ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
56422026-03-10T20:08:36.020ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
56432026-03-10T20:08:36.074ZTRACtest_retry_backoff (BackgroundTasks): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:52170/bfd/peers
56442026-03-10T20:08:36.075ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ec cd 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:9110 req_id = 779fcfb5-6553-46e1-9492-3567ac4d93b3
56452026-03-10T20:08:36.075ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60621,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:9110 req_id = 779fcfb5-6553-46e1-9492-3567ac4d93b3
56462026-03-10T20:08:36.075ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56472026-03-10T20:08:36.075ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:9110 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 779fcfb5-6553-46e1-9492-3567ac4d93b3
56482026-03-10T20:08:36.080ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
56492026-03-10T20:08:36.080ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
56502026-03-10T20:08:36.122ZTRACtest_retry_backoff (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:41467/switch/identifiers
56512026-03-10T20:08:36.123ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bff0735-78bb-4b12-8d72-54e8983b9293", "content-length": "2134", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
56522026-03-10T20:08:36.123ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet slot = 0 sp_type = sled
56532026-03-10T20:08:36.123ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = sp_ereport_ingester resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet slot = 0 sp_type = sled
56542026-03-10T20:08:36.123ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 0 sp_type = sled
56552026-03-10T20:08:36.123ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
56562026-03-10T20:08:36.123ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
56572026-03-10T20:08:36.125ZINFOtest_retry_backoff (dropshot_internal): accepted connection local_addr = 127.0.0.1:63060 remote_addr = 127.0.0.1:37558
56582026-03-10T20:08:36.125ZTRACtest_retry_backoff (BackgroundTasks): client response background_task = bfd_manager result = Ok(Response { url: "http://[::1]:52170/bfd/peers", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc9c6bb4-ebeb-4671-ae3d-b96689617797", "content-length": "2", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
56592026-03-10T20:08:36.127ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 641.877261ms iteration = 2 status = {}
56602026-03-10T20:08:36.127ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "9824fdb7-325d-4f3a-9e21-e6ceb8ae54a4", "content-length": "745", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
56612026-03-10T20:08:36.127ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet slot = 0 sp_type = switch
56622026-03-10T20:08:36.127ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = sp_ereport_ingester resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet slot = 0 sp_type = switch
56632026-03-10T20:08:36.127ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 0 sp_type = switch
56642026-03-10T20:08:36.127ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
56652026-03-10T20:08:36.128ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
56662026-03-10T20:08:36.129ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "b756e49f-e983-4926-9d0a-76ae5736487f", "content-length": "959", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
56672026-03-10T20:08:36.129ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet slot = 1 sp_type = sled
56682026-03-10T20:08:36.129ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = sp_ereport_ingester resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet slot = 1 sp_type = sled
56692026-03-10T20:08:36.129ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 1 sp_type = sled
56702026-03-10T20:08:36.129ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
56712026-03-10T20:08:36.129ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
56722026-03-10T20:08:36.130ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a2fb409-f5e0-4bc6-a730-336dc059bd6b", "content-length": "350", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
56732026-03-10T20:08:36.130ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet slot = 1 sp_type = switch
56742026-03-10T20:08:36.130ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = sp_ereport_ingester resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet slot = 1 sp_type = switch
56752026-03-10T20:08:36.130ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 1 sp_type = switch
56762026-03-10T20:08:36.130ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = switch
56772026-03-10T20:08:36.130ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = switch
56782026-03-10T20:08:36.130ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
56792026-03-10T20:08:36.131ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
56802026-03-10T20:08:36.134ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
56812026-03-10T20:08:36.134ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 0 sp_type = sled
56822026-03-10T20:08:36.134ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
56832026-03-10T20:08:36.134ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
56842026-03-10T20:08:36.135ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 04953021-2948-4b9a-9aef-c55f45435d1f uri = /nat/ipv4/changeset/0?limit=100
56852026-03-10T20:08:36.135ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 04953021-2948-4b9a-9aef-c55f45435d1f uri = /nat/ipv4/changeset/0?limit=100 version = 1.0.0
56862026-03-10T20:08:36.135ZTRACtest_retry_backoff (DpdClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Ok(Response { url: "http://[::1]:41467/switch/identifiers", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c0aabd9-bcaf-4f66-ad47-b46270fe3bdb", "content-length": "717", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
56872026-03-10T20:08:36.135ZTRACtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgd._tcp.control-plane.oxide.internal
56882026-03-10T20:08:36.136ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 04953021-2948-4b9a-9aef-c55f45435d1f resource = Database type = user_builtin uri = /nat/ipv4/changeset/0?limit=100 user_builtin_id = 001de000-05e4-4000-8000-000000000002
56892026-03-10T20:08:36.136ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 04953021-2948-4b9a-9aef-c55f45435d1f roles = RoleSet { roles: {} } type = user_builtin uri = /nat/ipv4/changeset/0?limit=100 user_builtin_id = 001de000-05e4-4000-8000-000000000002
56902026-03-10T20:08:36.136ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 04953021-2948-4b9a-9aef-c55f45435d1f resource = Database result = Ok(()) type = user_builtin uri = /nat/ipv4/changeset/0?limit=100 user_builtin_id = 001de000-05e4-4000-8000-000000000002
56912026-03-10T20:08:36.137ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
56922026-03-10T20:08:36.139ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
56932026-03-10T20:08:36.139ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 1 sp_type = sled
56942026-03-10T20:08:36.139ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
56952026-03-10T20:08:36.140ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
56962026-03-10T20:08:36.140ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = switch
56972026-03-10T20:08:36.143ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = switch
56982026-03-10T20:08:36.143ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 1 sp_type = switch
56992026-03-10T20:08:36.143ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = switch
57002026-03-10T20:08:36.143ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = switch
57012026-03-10T20:08:36.144ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
57022026-03-10T20:08:36.146ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
57032026-03-10T20:08:36.146ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database slot = 0 sp_type = switch
57042026-03-10T20:08:36.146ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
57052026-03-10T20:08:36.146ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
57062026-03-10T20:08:36.148ZTRACtest_retry_backoff (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5b 4c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 [L..........._mg\n0010: 64 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 d._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61191 req_id = 8fc91258-e77c-4554-9804-e53969a66dbb
57072026-03-10T20:08:36.148ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23372,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("_mgd._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgd._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 4,\n 95,\n 109,\n 103,\n 100,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:61191 req_id = 8fc91258-e77c-4554-9804-e53969a66dbb
57082026-03-10T20:08:36.148ZDEBGtest_retry_backoff (store): query key key = _mgd._tcp
57092026-03-10T20:08:36.148ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57102026-03-10T20:08:36.149ZDEBGtest_retry_backoff (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] kind = dns peer_addr = [::1]:61191 query = LowerQuery { name: LowerName(Name("_mgd._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgd._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgd._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 52170, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") }) }] req_id = 8fc91258-e77c-4554-9804-e53969a66dbb
57112026-03-10T20:08:36.150ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 16122 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 04953021-2948-4b9a-9aef-c55f45435d1f response_code = 200 uri = /nat/ipv4/changeset/0?limit=100
57122026-03-10T20:08:36.150ZDEBGtest_retry_backoff (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgd._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgd._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgd._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 52170, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }], valid_until: Instant { tv_sec: 6336, tv_nsec: 87439124 } })
57132026-03-10T20:08:36.150ZTRACtest_retry_backoff (DnsResolver): lookup_service_targets: looking up SRV target
57142026-03-10T20:08:36.152ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 16a93aef-5a33-420d-8ce4-e4a4717bc2d0 uri = /nat/ipv4/changeset/3?limit=100
57152026-03-10T20:08:36.152ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 16a93aef-5a33-420d-8ce4-e4a4717bc2d0 uri = /nat/ipv4/changeset/3?limit=100 version = 1.0.0
57162026-03-10T20:08:36.152ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 16a93aef-5a33-420d-8ce4-e4a4717bc2d0 resource = Database type = user_builtin uri = /nat/ipv4/changeset/3?limit=100 user_builtin_id = 001de000-05e4-4000-8000-000000000002
57172026-03-10T20:08:36.152ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 16a93aef-5a33-420d-8ce4-e4a4717bc2d0 roles = RoleSet { roles: {} } type = user_builtin uri = /nat/ipv4/changeset/3?limit=100 user_builtin_id = 001de000-05e4-4000-8000-000000000002
57182026-03-10T20:08:36.153ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 16a93aef-5a33-420d-8ce4-e4a4717bc2d0 resource = Database result = Ok(()) type = user_builtin uri = /nat/ipv4/changeset/3?limit=100 user_builtin_id = 001de000-05e4-4000-8000-000000000002
57192026-03-10T20:08:36.154ZTRACtest_retry_backoff (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 88 86 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:11817 req_id = 9e433116-9507-463a-a21f-1e5544efa2a7
57202026-03-10T20:08:36.154ZDEBGtest_retry_backoff (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34950,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} peer_addr = [::1]:11817 req_id = 9e433116-9507-463a-a21f-1e5544efa2a7
57212026-03-10T20:08:36.154ZDEBGtest_retry_backoff (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57222026-03-10T20:08:36.154ZDEBGtest_retry_backoff (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:11817 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(::1)) }] req_id = 9e433116-9507-463a-a21f-1e5544efa2a7
57232026-03-10T20:08:36.229ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database
57242026-03-10T20:08:36.229ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57252026-03-10T20:08:36.230ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57262026-03-10T20:08:36.231ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 84157 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:37558 req_id = 16a93aef-5a33-420d-8ce4-e4a4717bc2d0 response_code = 200 uri = /nat/ipv4/changeset/3?limit=100
57272026-03-10T20:08:36.232ZINFOtest_retry_backoff (BackgroundTasks): collected ereports from MGS background_task = sp_ereport_ingester ereports_received = 2 new_ereports = 2 received_restart_id = 0d3e464a-666e-4687-976f-90e31238be8b (ereporter_restart) requested_committed_ena = Some(Ena(0x2)) requested_restart_id = 0d3e464a-666e-4687-976f-90e31238be8b (ereporter_restart) requested_start_ena = Some(Ena(0x3)) slot = 0 sp_type = switch total_ereports_collected = 2 total_new_ereports = 2 total_requests = 1
57282026-03-10T20:08:36.232ZDEBGtest_retry_backoff (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = Some(Ena(0x2)) gateway_addr = [::1]:43524 restart_id = 0d3e464a-666e-4687-976f-90e31238be8b (ereporter_restart) slot = 0 sp_type = switch start_ena = Some(Ena(0x3))
57292026-03-10T20:08:36.232ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[::1]:43524 method = POST uri = http://[::1]:43524/sp/switch/0/ereports?committed=2&limit=255&restart_id=0d3e464a-666e-4687-976f-90e31238be8b&start_at=3
57302026-03-10T20:08:36.233ZINFOtest_retry_backoff (BackgroundTasks): collected ereports from MGS background_task = sp_ereport_ingester ereports_received = 1 new_ereports = 1 received_restart_id = c20c9921-e4fc-40d0-b341-92aa9e330717 (ereporter_restart) requested_committed_ena = Some(Ena(0x1)) requested_restart_id = c20c9921-e4fc-40d0-b341-92aa9e330717 (ereporter_restart) requested_start_ena = Some(Ena(0x2)) slot = 1 sp_type = switch total_ereports_collected = 1 total_new_ereports = 1 total_requests = 1
57312026-03-10T20:08:36.233ZDEBGtest_retry_backoff (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = Some(Ena(0x1)) gateway_addr = [::1]:43524 restart_id = c20c9921-e4fc-40d0-b341-92aa9e330717 (ereporter_restart) slot = 1 sp_type = switch start_ena = Some(Ena(0x2))
57322026-03-10T20:08:36.233ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[::1]:43524 method = POST uri = http://[::1]:43524/sp/switch/1/ereports?committed=1&limit=255&restart_id=c20c9921-e4fc-40d0-b341-92aa9e330717&start_at=2
57332026-03-10T20:08:36.233ZERROtest_retry_backoff (BackgroundTasks): no DPD client for switch location background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc switch_location = Switch1
57342026-03-10T20:08:36.234ZTRACtest_retry_backoff (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:41467/port/qsfp0/settings?tag=omicron
57352026-03-10T20:08:36.234ZINFOtest_retry_backoff (BackgroundTasks): collected ereports from MGS background_task = sp_ereport_ingester ereports_received = 6 new_ereports = 6 received_restart_id = af1ebf85-36ba-4c31-bbec-b9825d6d9d8b (ereporter_restart) requested_committed_ena = Some(Ena(0x6)) requested_restart_id = af1ebf85-36ba-4c31-bbec-b9825d6d9d8b (ereporter_restart) requested_start_ena = Some(Ena(0x7)) slot = 0 sp_type = sled total_ereports_collected = 6 total_new_ereports = 6 total_requests = 1
57362026-03-10T20:08:36.234ZDEBGtest_retry_backoff (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = Some(Ena(0x6)) gateway_addr = [::1]:43524 restart_id = af1ebf85-36ba-4c31-bbec-b9825d6d9d8b (ereporter_restart) slot = 0 sp_type = sled start_ena = Some(Ena(0x7))
57372026-03-10T20:08:36.234ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[::1]:43524 method = POST uri = http://[::1]:43524/sp/sled/0/ereports?committed=6&limit=255&restart_id=af1ebf85-36ba-4c31-bbec-b9825d6d9d8b&start_at=7
57382026-03-10T20:08:36.235ZTRACtest_retry_backoff (DpdClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Ok(Response { url: "http://[::1]:41467/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e8db223-7441-41a6-94f7-4f2935a2ec85", "content-length": "12", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
57392026-03-10T20:08:36.235ZINFOtest_retry_backoff (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc settings = PortSettings { links: {} } switch_port_id = Name(Name("qsfp0"))
57402026-03-10T20:08:36.235ZINFOtest_retry_backoff (BackgroundTasks): clearing switch port settings background_task = switch_port_config_manager port_id = Qsfp(Qsfp("qsfp0")) rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc switch_location = Switch0
57412026-03-10T20:08:36.235ZINFOtest_retry_backoff (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc switch_location = Switch0 switch_port_id = Name(Name("qsfp0"))
57422026-03-10T20:08:36.235ZTRACtest_retry_backoff (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:52170/static/route4
57432026-03-10T20:08:36.237ZTRACtest_retry_backoff (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Ok(Response { url: "http://[::1]:52170/static/route4", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5e2ee65-1490-41e9-a735-0f9deb89df61", "content-length": "2", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
57442026-03-10T20:08:36.237ZTRACtest_retry_backoff (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:52170/static/route6
57452026-03-10T20:08:36.237ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/0/ereports?committed=6&limit=255&restart_id=af1ebf85-36ba-4c31-bbec-b9825d6d9d8b&start_at=7", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcab64e4-1258-48ae-b6b9-7bd278a64ab5", "content-length": "81", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
57462026-03-10T20:08:36.237ZTRACtest_retry_backoff (BackgroundTasks): no ereports returned by SP background_task = sp_ereport_ingester committed_ena = Some(Ena(0x6)) restart_id = af1ebf85-36ba-4c31-bbec-b9825d6d9d8b (ereporter_restart) slot = 0 sp_type = sled start_ena = Some(Ena(0x7)) total_ereports_received = 6 total_new_ereports = 6
57472026-03-10T20:08:36.237ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/1/ereports?committed=1&limit=255&restart_id=c20c9921-e4fc-40d0-b341-92aa9e330717&start_at=2", status: 200, headers: {"content-type": "application/json", "x-request-id": "84d0d9bb-7265-4bf8-af76-fec0682c6c05", "content-length": "81", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
57482026-03-10T20:08:36.237ZTRACtest_retry_backoff (BackgroundTasks): no ereports returned by SP background_task = sp_ereport_ingester committed_ena = Some(Ena(0x1)) restart_id = c20c9921-e4fc-40d0-b341-92aa9e330717 (ereporter_restart) slot = 1 sp_type = switch start_ena = Some(Ena(0x2)) total_ereports_received = 1 total_new_ereports = 1
57492026-03-10T20:08:36.237ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/switch/0/ereports?committed=2&limit=255&restart_id=0d3e464a-666e-4687-976f-90e31238be8b&start_at=3", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5ef9116-96bf-4619-8c4e-c13a7cfdb17a", "content-length": "81", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
57502026-03-10T20:08:36.237ZTRACtest_retry_backoff (BackgroundTasks): no ereports returned by SP background_task = sp_ereport_ingester committed_ena = Some(Ena(0x2)) restart_id = 0d3e464a-666e-4687-976f-90e31238be8b (ereporter_restart) slot = 0 sp_type = switch start_ena = Some(Ena(0x3)) total_ereports_received = 2 total_new_ereports = 2
57512026-03-10T20:08:36.238ZTRACtest_retry_backoff (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Ok(Response { url: "http://[::1]:52170/static/route6", status: 200, headers: {"content-type": "application/json", "x-request-id": "1de52b5e-cf62-4286-b9ee-c600d1032d54", "content-length": "2", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
57522026-03-10T20:08:36.238ZINFOtest_retry_backoff (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc routes = {Switch0: {}}
57532026-03-10T20:08:36.238ZINFOtest_retry_backoff (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc routes = {}
57542026-03-10T20:08:36.238ZINFOtest_retry_backoff (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc routes = {}
57552026-03-10T20:08:36.238ZINFOtest_retry_backoff (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc routes = {}
57562026-03-10T20:08:36.238ZINFOtest_retry_backoff (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
57572026-03-10T20:08:36.238ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database
57582026-03-10T20:08:36.238ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57592026-03-10T20:08:36.238ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57602026-03-10T20:08:36.240ZINFOtest_retry_backoff (BackgroundTasks): collected ereports from MGS background_task = sp_ereport_ingester ereports_received = 2 new_ereports = 2 received_restart_id = 55e30cc7-a109-492f-aca9-735ed725df3c (ereporter_restart) requested_committed_ena = Some(Ena(0x2)) requested_restart_id = 55e30cc7-a109-492f-aca9-735ed725df3c (ereporter_restart) requested_start_ena = Some(Ena(0x3)) slot = 1 sp_type = sled total_ereports_collected = 2 total_new_ereports = 2 total_requests = 1
57612026-03-10T20:08:36.240ZDEBGtest_retry_backoff (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = Some(Ena(0x2)) gateway_addr = [::1]:43524 restart_id = 55e30cc7-a109-492f-aca9-735ed725df3c (ereporter_restart) slot = 1 sp_type = sled start_ena = Some(Ena(0x3))
57622026-03-10T20:08:36.240ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[::1]:43524 method = POST uri = http://[::1]:43524/sp/sled/1/ereports?committed=2&limit=255&restart_id=55e30cc7-a109-492f-aca9-735ed725df3c&start_at=3
57632026-03-10T20:08:36.242ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[::1]:43524 result = Ok(Response { url: "http://[::1]:43524/sp/sled/1/ereports?committed=2&limit=255&restart_id=55e30cc7-a109-492f-aca9-735ed725df3c&start_at=3", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6098908-bd9b-478e-b75b-1416cc0c07bb", "content-length": "81", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
57642026-03-10T20:08:36.242ZTRACtest_retry_backoff (BackgroundTasks): no ereports returned by SP background_task = sp_ereport_ingester committed_ena = Some(Ena(0x2)) restart_id = 55e30cc7-a109-492f-aca9-735ed725df3c (ereporter_restart) slot = 1 sp_type = sled start_ena = Some(Ena(0x3)) total_ereports_received = 2 total_new_ereports = 2
57652026-03-10T20:08:36.242ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 716.009325ms iteration = 2 status = {"disabled":false,"errors":[],"sps":[{"ereports_received":6,"errors":[],"new_ereports":6,"requests":2,"slot":0,"sp_type":"sled"},{"ereports_received":2,"errors":[],"new_ereports":2,"requests":2,"slot":1,"sp_type":"sled"},{"ereports_received":2,"errors":[],"new_ereports":2,"requests":2,"slot":0,"sp_type":"switch"},{"ereports_received":1,"errors":[],"new_ereports":1,"requests":2,"slot":1,"sp_type":"switch"}]}
57662026-03-10T20:08:36.243ZTRACtest_retry_backoff (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:41467/loopback/ipv4
57672026-03-10T20:08:36.244ZTRACtest_retry_backoff (DpdClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Ok(Response { url: "http://[::1]:41467/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ff48f3-5f15-47c9-a404-cb855f4d7c36", "content-length": "2", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
57682026-03-10T20:08:36.244ZTRACtest_retry_backoff (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:41467/loopback/ipv6
57692026-03-10T20:08:36.244ZTRACtest_retry_backoff (DpdClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Ok(Response { url: "http://[::1]:41467/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab90c0ad-1849-469c-b813-7ddb8f1a758b", "content-length": "2", "date": "Tue, 10 Mar 2026 20:08:36 GMT"} })
57702026-03-10T20:08:36.244ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database
57712026-03-10T20:08:36.244ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57722026-03-10T20:08:36.245ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57732026-03-10T20:08:36.248ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database
57742026-03-10T20:08:36.248ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57752026-03-10T20:08:36.249ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57762026-03-10T20:08:36.251ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database
57772026-03-10T20:08:36.251ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57782026-03-10T20:08:36.251ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57792026-03-10T20:08:36.255ZWARNtest_retry_backoff (BackgroundTasks): no bootstore config found in db background_task = switch_port_config_manager key = network_key rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
57802026-03-10T20:08:36.255ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database
57812026-03-10T20:08:36.255ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57822026-03-10T20:08:36.256ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57832026-03-10T20:08:36.264ZINFOtest_retry_backoff (BackgroundTasks): updating bootstore config background_task = switch_port_config_manager config = EarlyNetworkConfigBody { rack_network_config: RackNetworkConfig { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 0.0.0.0, infra_ip_last: 0.0.0.0, ports: [], bgp: [], bfd: [] } } rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
57842026-03-10T20:08:36.264ZDEBGtest_retry_backoff (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = PUT rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12345/network-bootstore-config
57852026-03-10T20:08:36.264ZDEBGtest_retry_backoff (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12345/network-bootstore-config", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:12345, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
57862026-03-10T20:08:36.264ZERROtest_retry_backoff (BackgroundTasks): error updating bootstore background_task = switch_port_config_manager error = Communication Error: error sending request for url (http://[::1]:12345/network-bootstore-config) location = switch0 rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc request = WriteNetworkConfigRequest { generation: 2, body: EarlyNetworkConfigBody { rack_network_config: RackNetworkConfig { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 0.0.0.0, infra_ip_last: 0.0.0.0, ports: [], bgp: [], bfd: [] } } }
57872026-03-10T20:08:36.266ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 786.060316ms iteration = 2 status = {}
57882026-03-10T20:08:36.365ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
57892026-03-10T20:08:36.365ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
57902026-03-10T20:08:36.367ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
57912026-03-10T20:08:36.367ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
57922026-03-10T20:08:36.367ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
57932026-03-10T20:08:36.367ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
57942026-03-10T20:08:36.902ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57952026-03-10T20:08:36.902ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57962026-03-10T20:08:36.902ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
57972026-03-10T20:08:36.903ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
57982026-03-10T20:08:36.905ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
57992026-03-10T20:08:36.905ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
58002026-03-10T20:08:36.907ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
58012026-03-10T20:08:36.907ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58022026-03-10T20:08:36.997ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
58032026-03-10T20:08:37.291ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
58042026-03-10T20:08:37.291ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58052026-03-10T20:08:37.291ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = b6ffbc2b-36c2-4f1d-8cae-44173862200e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58062026-03-10T20:08:37.291ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = b6ffbc2b-36c2-4f1d-8cae-44173862200e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
58072026-03-10T20:08:37.291ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = b6ffbc2b-36c2-4f1d-8cae-44173862200e resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
58082026-03-10T20:08:37.292ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = b6ffbc2b-36c2-4f1d-8cae-44173862200e roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
58092026-03-10T20:08:37.292ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = b6ffbc2b-36c2-4f1d-8cae-44173862200e resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
58102026-03-10T20:08:37.295ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3543 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = b6ffbc2b-36c2-4f1d-8cae-44173862200e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58112026-03-10T20:08:37.295ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6ffbc2b-36c2-4f1d-8cae-44173862200e", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:37 GMT"} })
58122026-03-10T20:08:37.295ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
58132026-03-10T20:08:37.295ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f8f9aac-bc85-487a-85f2-1627b3d9e3d5 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
58142026-03-10T20:08:37.295ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f8f9aac-bc85-487a-85f2-1627b3d9e3d5 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
58152026-03-10T20:08:37.295ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f8f9aac-bc85-487a-85f2-1627b3d9e3d5 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
58162026-03-10T20:08:37.296ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f8f9aac-bc85-487a-85f2-1627b3d9e3d5 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
58172026-03-10T20:08:37.296ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f8f9aac-bc85-487a-85f2-1627b3d9e3d5 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
58182026-03-10T20:08:37.298ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58192026-03-10T20:08:37.299ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58202026-03-10T20:08:37.299ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3980 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f8f9aac-bc85-487a-85f2-1627b3d9e3d5 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
58212026-03-10T20:08:37.299ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f8f9aac-bc85-487a-85f2-1627b3d9e3d5", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:37 GMT"} })
58222026-03-10T20:08:37.300ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
58232026-03-10T20:08:37.300ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58242026-03-10T20:08:37.300ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
58252026-03-10T20:08:37.301ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58262026-03-10T20:08:37.301ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58272026-03-10T20:08:37.301ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58282026-03-10T20:08:37.301ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58292026-03-10T20:08:37.840ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58302026-03-10T20:08:37.840ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58312026-03-10T20:08:37.840ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
58322026-03-10T20:08:37.841ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
58332026-03-10T20:08:37.843ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
58342026-03-10T20:08:37.843ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
58352026-03-10T20:08:37.844ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
58362026-03-10T20:08:37.844ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58372026-03-10T20:08:37.936ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
58382026-03-10T20:08:38.236ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58392026-03-10T20:08:38.236ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58402026-03-10T20:08:38.237ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58412026-03-10T20:08:38.238ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58422026-03-10T20:08:38.238ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58432026-03-10T20:08:38.238ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58442026-03-10T20:08:38.778ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58452026-03-10T20:08:38.778ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58462026-03-10T20:08:38.778ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
58472026-03-10T20:08:38.778ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
58482026-03-10T20:08:38.781ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
58492026-03-10T20:08:38.781ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
58502026-03-10T20:08:38.783ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
58512026-03-10T20:08:38.783ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58522026-03-10T20:08:38.873ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
58532026-03-10T20:08:39.165ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
58542026-03-10T20:08:39.165ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58552026-03-10T20:08:39.166ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0ee81eb3-e823-4b45-bbd3-fcb03f2840e1 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58562026-03-10T20:08:39.166ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0ee81eb3-e823-4b45-bbd3-fcb03f2840e1 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
58572026-03-10T20:08:39.166ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0ee81eb3-e823-4b45-bbd3-fcb03f2840e1 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
58582026-03-10T20:08:39.166ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0ee81eb3-e823-4b45-bbd3-fcb03f2840e1 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
58592026-03-10T20:08:39.166ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0ee81eb3-e823-4b45-bbd3-fcb03f2840e1 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
58602026-03-10T20:08:39.169ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3782 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0ee81eb3-e823-4b45-bbd3-fcb03f2840e1 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
58612026-03-10T20:08:39.169ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ee81eb3-e823-4b45-bbd3-fcb03f2840e1", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:39 GMT"} })
58622026-03-10T20:08:39.169ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
58632026-03-10T20:08:39.170ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7fc9c41a-6ccc-4824-bb96-9d7c0fa0df7f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
58642026-03-10T20:08:39.170ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7fc9c41a-6ccc-4824-bb96-9d7c0fa0df7f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
58652026-03-10T20:08:39.170ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7fc9c41a-6ccc-4824-bb96-9d7c0fa0df7f resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
58662026-03-10T20:08:39.170ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7fc9c41a-6ccc-4824-bb96-9d7c0fa0df7f roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
58672026-03-10T20:08:39.170ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7fc9c41a-6ccc-4824-bb96-9d7c0fa0df7f resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
58682026-03-10T20:08:39.173ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2997 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 7fc9c41a-6ccc-4824-bb96-9d7c0fa0df7f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
58692026-03-10T20:08:39.173ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fc9c41a-6ccc-4824-bb96-9d7c0fa0df7f", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:39 GMT"} })
58702026-03-10T20:08:39.173ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
58712026-03-10T20:08:39.173ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58722026-03-10T20:08:39.173ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
58732026-03-10T20:08:39.173ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58742026-03-10T20:08:39.173ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58752026-03-10T20:08:39.174ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58762026-03-10T20:08:39.175ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58772026-03-10T20:08:39.175ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58782026-03-10T20:08:39.175ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58792026-03-10T20:08:39.715ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58802026-03-10T20:08:39.715ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58812026-03-10T20:08:39.715ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
58822026-03-10T20:08:39.715ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
58832026-03-10T20:08:39.717ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
58842026-03-10T20:08:39.717ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
58852026-03-10T20:08:39.719ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
58862026-03-10T20:08:39.719ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58872026-03-10T20:08:39.809ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
58882026-03-10T20:08:40.111ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58892026-03-10T20:08:40.111ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58902026-03-10T20:08:40.112ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58912026-03-10T20:08:40.113ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58922026-03-10T20:08:40.113ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58932026-03-10T20:08:40.113ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
58942026-03-10T20:08:40.214ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = reconfigurator_config_watcher iteration = 8 reason = Timeout
58952026-03-10T20:08:40.214ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig
58962026-03-10T20:08:40.214ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = reconfigurator_config_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58972026-03-10T20:08:40.214ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
58982026-03-10T20:08:40.214ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
58992026-03-10T20:08:40.215ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
59002026-03-10T20:08:40.217ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59012026-03-10T20:08:40.221ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig result = Ok(())
59022026-03-10T20:08:40.221ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
59032026-03-10T20:08:40.221ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
59042026-03-10T20:08:40.222ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
59052026-03-10T20:08:40.224ZDEBGtest_retry_backoff (BackgroundTasks): reconfigurator config load complete background_task = reconfigurator_config_watcher switches = Loaded(ReconfiguratorConfigView { version: 1, config: ReconfiguratorConfig { planner_enabled: false, planner_config: PlannerConfig { add_zones_with_mupdate_override: false }, tuf_repo_pruner_enabled: true }, time_modified: 2026-03-10T20:08:05.458980Z })
59062026-03-10T20:08:40.224ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = reconfigurator_config_watcher elapsed = 10.16405ms iteration = 8 status = {"config_updated":false}
59072026-03-10T20:08:40.308ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
59082026-03-10T20:08:40.309ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = df0744d9-4f0b-4179-90a5-b4c027bf9ad8 uri = /bgtasks/view/webhook_deliverator
59092026-03-10T20:08:40.309ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = df0744d9-4f0b-4179-90a5-b4c027bf9ad8 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59102026-03-10T20:08:40.309ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = df0744d9-4f0b-4179-90a5-b4c027bf9ad8 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59112026-03-10T20:08:40.309ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = df0744d9-4f0b-4179-90a5-b4c027bf9ad8 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59122026-03-10T20:08:40.309ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = df0744d9-4f0b-4179-90a5-b4c027bf9ad8 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59132026-03-10T20:08:40.309ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = df0744d9-4f0b-4179-90a5-b4c027bf9ad8 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59142026-03-10T20:08:40.312ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = df0744d9-4f0b-4179-90a5-b4c027bf9ad8 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59152026-03-10T20:08:40.315ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = df0744d9-4f0b-4179-90a5-b4c027bf9ad8 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59162026-03-10T20:08:40.315ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 6833 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = df0744d9-4f0b-4179-90a5-b4c027bf9ad8 response_code = 200 uri = /bgtasks/view/webhook_deliverator
59172026-03-10T20:08:40.315ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
59182026-03-10T20:08:40.315ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:50070/bgtasks/activate
59192026-03-10T20:08:40.316ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = db3b6120-ed51-4e68-9702-ac4632077a65 uri = /bgtasks/activate
59202026-03-10T20:08:40.316ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = db3b6120-ed51-4e68-9702-ac4632077a65 resource = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
59212026-03-10T20:08:40.316ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = db3b6120-ed51-4e68-9702-ac4632077a65 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
59222026-03-10T20:08:40.316ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = db3b6120-ed51-4e68-9702-ac4632077a65 resource = Database type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
59232026-03-10T20:08:40.316ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = db3b6120-ed51-4e68-9702-ac4632077a65 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
59242026-03-10T20:08:40.316ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = db3b6120-ed51-4e68-9702-ac4632077a65 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
59252026-03-10T20:08:40.318ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = db3b6120-ed51-4e68-9702-ac4632077a65 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
59262026-03-10T20:08:40.320ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = db3b6120-ed51-4e68-9702-ac4632077a65 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
59272026-03-10T20:08:40.320ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = webhook_deliverator iteration = 7 reason = Signaled
59282026-03-10T20:08:40.321ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database
59292026-03-10T20:08:40.321ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
59302026-03-10T20:08:40.321ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(())
59312026-03-10T20:08:40.321ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 6231 local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = db3b6120-ed51-4e68-9702-ac4632077a65 response_code = 204 uri = /bgtasks/activate
59322026-03-10T20:08:40.322ZINFOtest_retry_backoff (lockstep client test context): client received response status = 204
59332026-03-10T20:08:40.322ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
59342026-03-10T20:08:40.322ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 64f0356c-df7f-46d4-ac66-b5ae4d5175b2 uri = /bgtasks/view/webhook_deliverator
59352026-03-10T20:08:40.323ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 64f0356c-df7f-46d4-ac66-b5ae4d5175b2 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59362026-03-10T20:08:40.323ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 64f0356c-df7f-46d4-ac66-b5ae4d5175b2 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59372026-03-10T20:08:40.323ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 64f0356c-df7f-46d4-ac66-b5ae4d5175b2 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59382026-03-10T20:08:40.323ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 64f0356c-df7f-46d4-ac66-b5ae4d5175b2 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59392026-03-10T20:08:40.323ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 64f0356c-df7f-46d4-ac66-b5ae4d5175b2 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59402026-03-10T20:08:40.324ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 64f0356c-df7f-46d4-ac66-b5ae4d5175b2 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59412026-03-10T20:08:40.327ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 64f0356c-df7f-46d4-ac66-b5ae4d5175b2 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59422026-03-10T20:08:40.327ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 5678 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = 64f0356c-df7f-46d4-ac66-b5ae4d5175b2 response_code = 200 uri = /bgtasks/view/webhook_deliverator
59432026-03-10T20:08:40.328ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
59442026-03-10T20:08:40.328ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
59452026-03-10T20:08:40.328ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
59462026-03-10T20:08:40.329ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
59472026-03-10T20:08:40.331ZINFOtest_retry_backoff (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator num_receivers = 1
59482026-03-10T20:08:40.331ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 11.32415ms iteration = 7 status = {"by_rx":{"2b6fbbc6-d38e-4d46-a5cd-293e75593af0":{"already_delivered":0,"delivered_ok":0,"delivery_errors":{},"error":null,"failed_deliveries":[],"in_progress":0,"ready":0}},"error":null}
59492026-03-10T20:08:40.376ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
59502026-03-10T20:08:40.376ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d6b047df-0c4c-454b-bd43-e81a033a69c1 uri = /bgtasks/view/webhook_deliverator
59512026-03-10T20:08:40.376ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d6b047df-0c4c-454b-bd43-e81a033a69c1 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59522026-03-10T20:08:40.376ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d6b047df-0c4c-454b-bd43-e81a033a69c1 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59532026-03-10T20:08:40.376ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d6b047df-0c4c-454b-bd43-e81a033a69c1 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59542026-03-10T20:08:40.376ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d6b047df-0c4c-454b-bd43-e81a033a69c1 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59552026-03-10T20:08:40.377ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d6b047df-0c4c-454b-bd43-e81a033a69c1 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59562026-03-10T20:08:40.379ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d6b047df-0c4c-454b-bd43-e81a033a69c1 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59572026-03-10T20:08:40.383ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d6b047df-0c4c-454b-bd43-e81a033a69c1 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
59582026-03-10T20:08:40.383ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 7231 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d6b047df-0c4c-454b-bd43-e81a033a69c1 response_code = 200 uri = /bgtasks/view/webhook_deliverator
59592026-03-10T20:08:40.383ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
59602026-03-10T20:08:40.652ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
59612026-03-10T20:08:40.652ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
59622026-03-10T20:08:40.652ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
59632026-03-10T20:08:40.653ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
59642026-03-10T20:08:40.655ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
59652026-03-10T20:08:40.655ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
59662026-03-10T20:08:40.656ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
59672026-03-10T20:08:40.656ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
59682026-03-10T20:08:40.747ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
59692026-03-10T20:08:41.040ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
59702026-03-10T20:08:41.041ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
59712026-03-10T20:08:41.041ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 237e228b-4d9e-4a4d-a04e-8cf5a0c07370 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
59722026-03-10T20:08:41.041ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 237e228b-4d9e-4a4d-a04e-8cf5a0c07370 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
59732026-03-10T20:08:41.041ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 237e228b-4d9e-4a4d-a04e-8cf5a0c07370 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
59742026-03-10T20:08:41.041ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 237e228b-4d9e-4a4d-a04e-8cf5a0c07370 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
59752026-03-10T20:08:41.042ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 237e228b-4d9e-4a4d-a04e-8cf5a0c07370 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
59762026-03-10T20:08:41.044ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3323 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 237e228b-4d9e-4a4d-a04e-8cf5a0c07370 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
59772026-03-10T20:08:41.044ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "237e228b-4d9e-4a4d-a04e-8cf5a0c07370", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:41 GMT"} })
59782026-03-10T20:08:41.044ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
59792026-03-10T20:08:41.045ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 992aa279-871a-417c-9b74-a474e3750a45 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
59802026-03-10T20:08:41.045ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 992aa279-871a-417c-9b74-a474e3750a45 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
59812026-03-10T20:08:41.045ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 992aa279-871a-417c-9b74-a474e3750a45 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
59822026-03-10T20:08:41.045ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 992aa279-871a-417c-9b74-a474e3750a45 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
59832026-03-10T20:08:41.045ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 992aa279-871a-417c-9b74-a474e3750a45 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
59842026-03-10T20:08:41.047ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2662 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 992aa279-871a-417c-9b74-a474e3750a45 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
59852026-03-10T20:08:41.047ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "992aa279-871a-417c-9b74-a474e3750a45", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:41 GMT"} })
59862026-03-10T20:08:41.047ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
59872026-03-10T20:08:41.047ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59882026-03-10T20:08:41.047ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
59892026-03-10T20:08:41.049ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
59902026-03-10T20:08:41.049ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
59912026-03-10T20:08:41.049ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
59922026-03-10T20:08:41.049ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
59932026-03-10T20:08:41.049ZTRACtest_retry_backoff (dropshot): incoming request local_addr = 127.0.0.1:33574 method = GET remote_addr = 127.0.0.1:51522 req_id = 6b3d9384-eabc-43c2-aa71-2ef773062b32 uri = /913233fe-92a8-4635-9572-183f495429c4
59942026-03-10T20:08:41.055ZINFOtest_retry_backoff (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867 latency_us = 6061 local_addr = 127.0.0.1:33574 method = GET remote_addr = 127.0.0.1:51522 req_id = 6b3d9384-eabc-43c2-aa71-2ef773062b32 response_code = 200 uri = /913233fe-92a8-4635-9572-183f495429c4
59952026-03-10T20:08:41.055ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
59962026-03-10T20:08:41.061ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
59972026-03-10T20:08:41.061ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
59982026-03-10T20:08:41.061ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4 producer_ip = 127.0.0.1 producer_port = 33574
59992026-03-10T20:08:41.061ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60002026-03-10T20:08:41.061ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60012026-03-10T20:08:41.061ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60022026-03-10T20:08:41.061ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60032026-03-10T20:08:41.061ZDEBGtest_retry_backoff (results-sink): inserting 10 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60042026-03-10T20:08:41.061ZDEBGtest_retry_backoff (results-sink-cluster): inserting 10 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60052026-03-10T20:08:41.061ZDEBGtest_retry_backoff (clickhouse-client): unrolling 10 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
60062026-03-10T20:08:41.066ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 20 sql = INSERT INTO oximeter.fields_string FORMAT Native
60072026-03-10T20:08:41.068ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 20 table_name = fields_string
60082026-03-10T20:08:41.068ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
60092026-03-10T20:08:41.069ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 10 table_name = fields_u16
60102026-03-10T20:08:41.069ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
60112026-03-10T20:08:41.070ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 10 table_name = fields_uuid
60122026-03-10T20:08:41.071ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 19 n_rows = 10 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
60132026-03-10T20:08:41.073ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 10 table_name = measurements_histogramu64
60142026-03-10T20:08:41.073ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60152026-03-10T20:08:41.156ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
60162026-03-10T20:08:41.590ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60172026-03-10T20:08:41.590ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60182026-03-10T20:08:41.590ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
60192026-03-10T20:08:41.591ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
60202026-03-10T20:08:41.593ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
60212026-03-10T20:08:41.593ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
60222026-03-10T20:08:41.594ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
60232026-03-10T20:08:41.595ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60242026-03-10T20:08:41.685ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
60252026-03-10T20:08:41.986ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60262026-03-10T20:08:41.987ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60272026-03-10T20:08:41.988ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60282026-03-10T20:08:41.988ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60292026-03-10T20:08:41.988ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60302026-03-10T20:08:41.988ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60312026-03-10T20:08:42.527ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60322026-03-10T20:08:42.527ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60332026-03-10T20:08:42.527ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
60342026-03-10T20:08:42.528ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
60352026-03-10T20:08:42.530ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
60362026-03-10T20:08:42.530ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
60372026-03-10T20:08:42.531ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
60382026-03-10T20:08:42.531ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60392026-03-10T20:08:42.622ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
60402026-03-10T20:08:42.915ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
60412026-03-10T20:08:42.916ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60422026-03-10T20:08:42.916ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 92ed2e7b-d918-4669-9b66-68324e25c785 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60432026-03-10T20:08:42.916ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 92ed2e7b-d918-4669-9b66-68324e25c785 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
60442026-03-10T20:08:42.916ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 92ed2e7b-d918-4669-9b66-68324e25c785 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
60452026-03-10T20:08:42.916ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 92ed2e7b-d918-4669-9b66-68324e25c785 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
60462026-03-10T20:08:42.917ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 92ed2e7b-d918-4669-9b66-68324e25c785 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
60472026-03-10T20:08:42.920ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 4124 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 92ed2e7b-d918-4669-9b66-68324e25c785 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60482026-03-10T20:08:42.920ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "92ed2e7b-d918-4669-9b66-68324e25c785", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:42 GMT"} })
60492026-03-10T20:08:42.920ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60502026-03-10T20:08:42.921ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a194a1c7-544e-45e7-ae05-a2bbcf61bd82 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60512026-03-10T20:08:42.921ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a194a1c7-544e-45e7-ae05-a2bbcf61bd82 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
60522026-03-10T20:08:42.921ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a194a1c7-544e-45e7-ae05-a2bbcf61bd82 resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
60532026-03-10T20:08:42.921ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a194a1c7-544e-45e7-ae05-a2bbcf61bd82 roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
60542026-03-10T20:08:42.921ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a194a1c7-544e-45e7-ae05-a2bbcf61bd82 resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
60552026-03-10T20:08:42.923ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60562026-03-10T20:08:42.923ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60572026-03-10T20:08:42.923ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3115 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = a194a1c7-544e-45e7-ae05-a2bbcf61bd82 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60582026-03-10T20:08:42.924ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "a194a1c7-544e-45e7-ae05-a2bbcf61bd82", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:42 GMT"} })
60592026-03-10T20:08:42.924ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
60602026-03-10T20:08:42.924ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60612026-03-10T20:08:42.924ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
60622026-03-10T20:08:42.925ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60632026-03-10T20:08:42.925ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60642026-03-10T20:08:42.925ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60652026-03-10T20:08:42.925ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60662026-03-10T20:08:43.465ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60672026-03-10T20:08:43.465ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60682026-03-10T20:08:43.465ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
60692026-03-10T20:08:43.466ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
60702026-03-10T20:08:43.468ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
60712026-03-10T20:08:43.468ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
60722026-03-10T20:08:43.469ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
60732026-03-10T20:08:43.469ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60742026-03-10T20:08:43.561ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
60752026-03-10T20:08:43.861ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60762026-03-10T20:08:43.861ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60772026-03-10T20:08:43.862ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60782026-03-10T20:08:43.863ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60792026-03-10T20:08:43.863ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60802026-03-10T20:08:43.863ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
60812026-03-10T20:08:44.012ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:48341 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
60822026-03-10T20:08:44.013ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:48341 req_id = 5e8c54f8-0350-405e-8608-74ec556da665 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
60832026-03-10T20:08:44.014ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 1561 local_addr = [::1]:36870 method = GET remote_addr = [::1]:48341 req_id = 5e8c54f8-0350-405e-8608-74ec556da665 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
60842026-03-10T20:08:44.399ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:44631 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
60852026-03-10T20:08:44.399ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:44631 req_id = 659c6886-3a65-40c3-a278-c7a3528766c3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
60862026-03-10T20:08:44.399ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 318 local_addr = [::1]:36870 method = GET remote_addr = [::1]:44631 req_id = 659c6886-3a65-40c3-a278-c7a3528766c3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
60872026-03-10T20:08:44.399ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:32814 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
60882026-03-10T20:08:44.400ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:32814 req_id = da1bbc12-87ae-445f-b1f7-0c3beb61b260 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
60892026-03-10T20:08:44.400ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 66 local_addr = [::1]:36870 method = GET remote_addr = [::1]:32814 req_id = da1bbc12-87ae-445f-b1f7-0c3beb61b260 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
60902026-03-10T20:08:44.400ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:39372 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
60912026-03-10T20:08:44.400ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:49857 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
60922026-03-10T20:08:44.400ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:39372 req_id = ed4f3d12-9d5c-4d0a-8b87-7505c99455a3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
60932026-03-10T20:08:44.400ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:49857 req_id = 7fe4ea18-49ae-4d62-8f1b-8d21e7d9ffa2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
60942026-03-10T20:08:44.400ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 128 local_addr = [::1]:36870 method = GET remote_addr = [::1]:39372 req_id = ed4f3d12-9d5c-4d0a-8b87-7505c99455a3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
60952026-03-10T20:08:44.400ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 92 local_addr = [::1]:36870 method = GET remote_addr = [::1]:49857 req_id = 7fe4ea18-49ae-4d62-8f1b-8d21e7d9ffa2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
60962026-03-10T20:08:44.403ZDEBGtest_retry_backoff (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60972026-03-10T20:08:44.403ZDEBGtest_retry_backoff (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60982026-03-10T20:08:44.403ZDEBGtest_retry_backoff (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f
60992026-03-10T20:08:44.403ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
61002026-03-10T20:08:44.404ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:56673 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
61012026-03-10T20:08:44.404ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:56673 req_id = 70b9b1c7-0e8d-4563-876b-c26420af2826 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
61022026-03-10T20:08:44.404ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 99 local_addr = [::1]:36870 method = GET remote_addr = [::1]:56673 req_id = 70b9b1c7-0e8d-4563-876b-c26420af2826 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
61032026-03-10T20:08:44.405ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 2 table_name = fields_string
61042026-03-10T20:08:44.406ZTRACtest_retry_backoff (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
61052026-03-10T20:08:44.407ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:64926 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
61062026-03-10T20:08:44.407ZDEBGtest_retry_backoff (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 8067c9f8-2c90-409c-98ac-9b1d080d115f n_rows = 1 table_name = measurements_i64
61072026-03-10T20:08:44.407ZTRACtest_retry_backoff (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
61082026-03-10T20:08:44.407ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:64926 req_id = c7de8ae0-eb4e-480b-8cbf-8d9b031b05dc sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
61092026-03-10T20:08:44.407ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 103 local_addr = [::1]:36870 method = GET remote_addr = [::1]:64926 req_id = c7de8ae0-eb4e-480b-8cbf-8d9b031b05dc response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
61102026-03-10T20:08:44.411ZINFOtest_retry_backoff (dropshot): accepted connection kind = pantry local_addr = [::1]:36870 remote_addr = [::1]:46067 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
61112026-03-10T20:08:44.411ZTRACtest_retry_backoff (dropshot): incoming request kind = pantry local_addr = [::1]:36870 method = GET remote_addr = [::1]:46067 req_id = ee3edcc3-afe2-45bd-b528-08a7e504b10d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
61122026-03-10T20:08:44.411ZINFOtest_retry_backoff (dropshot): request completed kind = pantry latency_us = 82 local_addr = [::1]:36870 method = GET remote_addr = [::1]:46067 req_id = ee3edcc3-afe2-45bd-b528-08a7e504b10d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
61132026-03-10T20:08:44.498ZWARNtest_retry_backoff (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
61142026-03-10T20:08:44.790ZINFOtest_retry_backoff (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:557
61152026-03-10T20:08:44.791ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
61162026-03-10T20:08:44.791ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4fed0e51-62d4-46f6-ae27-50a8d939c9cc uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
61172026-03-10T20:08:44.791ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4fed0e51-62d4-46f6-ae27-50a8d939c9cc uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending version = 1.0.0
61182026-03-10T20:08:44.791ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4fed0e51-62d4-46f6-ae27-50a8d939c9cc resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
61192026-03-10T20:08:44.791ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4fed0e51-62d4-46f6-ae27-50a8d939c9cc roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
61202026-03-10T20:08:44.792ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4fed0e51-62d4-46f6-ae27-50a8d939c9cc resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending user_builtin_id = 001de000-05e4-4000-8000-000000000002
61212026-03-10T20:08:44.794ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 3740 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 4fed0e51-62d4-46f6-ae27-50a8d939c9cc response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
61222026-03-10T20:08:44.795ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fed0e51-62d4-46f6-ae27-50a8d939c9cc", "content-length": "405", "date": "Tue, 10 Mar 2026 20:08:44 GMT"} })
61232026-03-10T20:08:44.795ZDEBGtest_retry_backoff (oximeter): client request body = None method = GET uri = http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
61242026-03-10T20:08:44.795ZTRACtest_retry_backoff (dropshot_internal): incoming request local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f6c2a68-3af1-4325-83c6-4a74018010eb uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
61252026-03-10T20:08:44.795ZDEBGtest_retry_backoff (dropshot_internal): determined request API version local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f6c2a68-3af1-4325-83c6-4a74018010eb uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D version = 1.0.0
61262026-03-10T20:08:44.795ZTRACtest_retry_backoff (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f6c2a68-3af1-4325-83c6-4a74018010eb resource = Database type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
61272026-03-10T20:08:44.795ZDEBGtest_retry_backoff (dropshot_internal): roles authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f6c2a68-3af1-4325-83c6-4a74018010eb roles = RoleSet { roles: {} } type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
61282026-03-10T20:08:44.796ZDEBGtest_retry_backoff (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f6c2a68-3af1-4325-83c6-4a74018010eb resource = Database result = Ok(()) type = user_builtin uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D user_builtin_id = 001de000-05e4-4000-8000-000000000002
61292026-03-10T20:08:44.797ZINFOtest_retry_backoff (dropshot_internal): request completed latency_us = 2416 local_addr = 127.0.0.1:63060 method = GET remote_addr = 127.0.0.1:44916 req_id = 0f6c2a68-3af1-4325-83c6-4a74018010eb response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
61302026-03-10T20:08:44.797ZDEBGtest_retry_backoff (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:63060/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f6c2a68-3af1-4325-83c6-4a74018010eb", "content-length": "29", "date": "Tue, 10 Mar 2026 20:08:44 GMT"} })
61312026-03-10T20:08:44.797ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 913233fe-92a8-4635-9572-183f495429c4
61322026-03-10T20:08:44.797ZTRACtest_retry_backoff (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61332026-03-10T20:08:44.797ZINFOtest_retry_backoff (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:632 n_current_tasks = 2 n_pruned_tasks = 0
61342026-03-10T20:08:44.799ZDEBGtest_retry_backoff (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
61352026-03-10T20:08:44.799ZDEBGtest_retry_backoff (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
61362026-03-10T20:08:44.800ZTRACtest_retry_backoff (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
61372026-03-10T20:08:44.800ZDEBGtest_retry_backoff (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
61382026-03-10T20:08:44.800ZTRACtest_retry_backoff (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
61392026-03-10T20:08:44.800ZTRACtest_retry_backoff (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de producer_ip = ::1 producer_port = 41614
61402026-03-10T20:08:44.902ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = reconfigurator_config_watcher iteration = 9 reason = Timeout
61412026-03-10T20:08:44.902ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig
61422026-03-10T20:08:44.902ZTRACtest_retry_backoff (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } background_task = reconfigurator_config_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61432026-03-10T20:08:44.902ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
61442026-03-10T20:08:44.902ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
61452026-03-10T20:08:44.903ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
61462026-03-10T20:08:44.905ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61472026-03-10T20:08:44.909ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = BlueprintConfig result = Ok(())
61482026-03-10T20:08:44.909ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database
61492026-03-10T20:08:44.909ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = reconfigurator_config_watcher roles = RoleSet { roles: {} }
61502026-03-10T20:08:44.909ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = reconfigurator_config_watcher resource = Database result = Ok(())
61512026-03-10T20:08:44.911ZDEBGtest_retry_backoff (BackgroundTasks): reconfigurator config load complete background_task = reconfigurator_config_watcher switches = Loaded(ReconfiguratorConfigView { version: 1, config: ReconfiguratorConfig { planner_enabled: false, planner_config: PlannerConfig { add_zones_with_mupdate_override: false }, tuf_repo_pruner_enabled: true }, time_modified: 2026-03-10T20:08:05.458980Z })
61522026-03-10T20:08:44.911ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = reconfigurator_config_watcher elapsed = 10.245041ms iteration = 9 status = {"config_updated":false}
61532026-03-10T20:08:45.073ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
61542026-03-10T20:08:45.073ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = a938616b-f23d-45be-add1-255e0526c865 uri = /bgtasks/view/webhook_deliverator
61552026-03-10T20:08:45.073ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = a938616b-f23d-45be-add1-255e0526c865 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61562026-03-10T20:08:45.073ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = a938616b-f23d-45be-add1-255e0526c865 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61572026-03-10T20:08:45.073ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = a938616b-f23d-45be-add1-255e0526c865 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61582026-03-10T20:08:45.073ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = a938616b-f23d-45be-add1-255e0526c865 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61592026-03-10T20:08:45.074ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = a938616b-f23d-45be-add1-255e0526c865 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61602026-03-10T20:08:45.076ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = a938616b-f23d-45be-add1-255e0526c865 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61612026-03-10T20:08:45.080ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = a938616b-f23d-45be-add1-255e0526c865 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61622026-03-10T20:08:45.080ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 7350 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = a938616b-f23d-45be-add1-255e0526c865 response_code = 200 uri = /bgtasks/view/webhook_deliverator
61632026-03-10T20:08:45.080ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
61642026-03-10T20:08:45.081ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:50070/bgtasks/activate
61652026-03-10T20:08:45.081ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 424d708e-c2b6-4f61-af43-b73b6e969bae uri = /bgtasks/activate
61662026-03-10T20:08:45.081ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 424d708e-c2b6-4f61-af43-b73b6e969bae resource = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
61672026-03-10T20:08:45.081ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 424d708e-c2b6-4f61-af43-b73b6e969bae resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
61682026-03-10T20:08:45.081ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 424d708e-c2b6-4f61-af43-b73b6e969bae resource = Database type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
61692026-03-10T20:08:45.081ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 424d708e-c2b6-4f61-af43-b73b6e969bae roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
61702026-03-10T20:08:45.081ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 424d708e-c2b6-4f61-af43-b73b6e969bae resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
61712026-03-10T20:08:45.084ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 424d708e-c2b6-4f61-af43-b73b6e969bae roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
61722026-03-10T20:08:45.086ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 424d708e-c2b6-4f61-af43-b73b6e969bae resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/activate user_builtin_id = 001de000-05e4-4000-8000-000000000002
61732026-03-10T20:08:45.086ZDEBGtest_retry_backoff (BackgroundTasks): activating background_task = webhook_deliverator iteration = 8 reason = Signaled
61742026-03-10T20:08:45.086ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database
61752026-03-10T20:08:45.086ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
61762026-03-10T20:08:45.087ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(())
61772026-03-10T20:08:45.087ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 7061 local_addr = 127.0.0.1:50070 method = POST remote_addr = 127.0.0.1:43636 req_id = 424d708e-c2b6-4f61-af43-b73b6e969bae response_code = 204 uri = /bgtasks/activate
61782026-03-10T20:08:45.088ZINFOtest_retry_backoff (lockstep client test context): client received response status = 204
61792026-03-10T20:08:45.088ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
61802026-03-10T20:08:45.088ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d8ca0fc4-0676-4aeb-bd3e-d441ab9f50fa uri = /bgtasks/view/webhook_deliverator
61812026-03-10T20:08:45.088ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d8ca0fc4-0676-4aeb-bd3e-d441ab9f50fa resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61822026-03-10T20:08:45.088ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d8ca0fc4-0676-4aeb-bd3e-d441ab9f50fa resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61832026-03-10T20:08:45.088ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d8ca0fc4-0676-4aeb-bd3e-d441ab9f50fa resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61842026-03-10T20:08:45.088ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d8ca0fc4-0676-4aeb-bd3e-d441ab9f50fa roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61852026-03-10T20:08:45.088ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d8ca0fc4-0676-4aeb-bd3e-d441ab9f50fa resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61862026-03-10T20:08:45.090ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d8ca0fc4-0676-4aeb-bd3e-d441ab9f50fa roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61872026-03-10T20:08:45.093ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d8ca0fc4-0676-4aeb-bd3e-d441ab9f50fa resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61882026-03-10T20:08:45.094ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 6026 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = d8ca0fc4-0676-4aeb-bd3e-d441ab9f50fa response_code = 200 uri = /bgtasks/view/webhook_deliverator
61892026-03-10T20:08:45.094ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200
61902026-03-10T20:08:45.096ZTRACtest_retry_backoff (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
61912026-03-10T20:08:45.096ZDEBGtest_retry_backoff (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} } rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
61922026-03-10T20:08:45.096ZDEBGtest_retry_backoff (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) background_task = webhook_deliverator resource = Database result = Ok(()) rx_id = 2b6fbbc6-d38e-4d46-a5cd-293e75593af0 rx_name = my-great-webhook
61932026-03-10T20:08:45.099ZINFOtest_retry_backoff (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator num_receivers = 1
61942026-03-10T20:08:45.099ZDEBGtest_retry_backoff (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 13.328597ms iteration = 8 status = {"by_rx":{"2b6fbbc6-d38e-4d46-a5cd-293e75593af0":{"already_delivered":0,"delivered_ok":0,"delivery_errors":{},"error":null,"failed_deliveries":[],"in_progress":0,"ready":0}},"error":null}
61952026-03-10T20:08:45.142ZINFOtest_retry_backoff (lockstep client test context): client request body = Body { inner: BoxBody } method = GET uri = http://127.0.0.1:50070/bgtasks/view/webhook_deliverator
61962026-03-10T20:08:45.143ZTRACtest_retry_backoff (dropshot_lockstep): incoming request local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = b89ed472-4019-47a8-8889-c609aeb904b0 uri = /bgtasks/view/webhook_deliverator
61972026-03-10T20:08:45.143ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = b89ed472-4019-47a8-8889-c609aeb904b0 resource = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61982026-03-10T20:08:45.143ZTRACtest_retry_backoff (dropshot_lockstep): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. } authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = b89ed472-4019-47a8-8889-c609aeb904b0 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
61992026-03-10T20:08:45.143ZTRACtest_retry_backoff (dropshot_lockstep): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = b89ed472-4019-47a8-8889-c609aeb904b0 resource = Database type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
62002026-03-10T20:08:45.143ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = b89ed472-4019-47a8-8889-c609aeb904b0 roles = RoleSet { roles: {} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
62012026-03-10T20:08:45.144ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = b89ed472-4019-47a8-8889-c609aeb904b0 resource = Database result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
62022026-03-10T20:08:45.146ZDEBGtest_retry_backoff (dropshot_lockstep): roles authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = b89ed472-4019-47a8-8889-c609aeb904b0 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
62032026-03-10T20:08:45.149ZDEBGtest_retry_backoff (dropshot_lockstep): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. }) authenticated = true local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = b89ed472-4019-47a8-8889-c609aeb904b0 resource = Fleet result = Ok(()) type = user_builtin uri = /bgtasks/view/webhook_deliverator user_builtin_id = 001de000-05e4-4000-8000-000000000002
62042026-03-10T20:08:45.149ZINFOtest_retry_backoff (dropshot_lockstep): request completed latency_us = 7085 local_addr = 127.0.0.1:50070 method = GET remote_addr = 127.0.0.1:43636 req_id = b89ed472-4019-47a8-8889-c609aeb904b0 response_code = 200 uri = /bgtasks/view/webhook_deliverator
62052026-03-10T20:08:45.150ZINFOtest_retry_backoff (lockstep client test context): client received response status = 200