Max level shown:
12025-04-30T06:38:02.704ZDEBGtest_quotas (init_with_steps): Running step start_crdb
22025-04-30T06:38:02.706ZDEBGtest_quotas: Starting CRDB
32025-04-30T06:38:02.742ZINFOtest_quotas: cockroach temporary directory: /var/tmp/omicron_tmp/.tmpJZg4ZN
42025-04-30T06:38:02.742ZINFOtest_quotas: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmpJZg4ZN/data)
52025-04-30T06:38:02.760ZINFOtest_quotas: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmpJZg4ZN/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmpJZg4ZN/listen-url
62025-04-30T06:38:02.761ZINFOtest_quotas: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=2 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:38:04.096ZINFOtest_quotas: cockroach pid: 25431
82025-04-30T06:38:04.096ZINFOtest_quotas: cockroach listen URL: postgresql://root@[::1]:61168/omicron?sslmode=disable
92025-04-30T06:38:04.097ZDEBGtest_quotas (init_with_steps): Running step start_clickhouse
102025-04-30T06:38:04.097ZDEBGtest_quotas: Starting Clickhouse
112025-04-30T06:38:04.821ZDEBGtest_quotas (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:38:04.821ZDEBGtest_quotas: Starting Management Gateway
132025-04-30T06:38:04.948ZDEBGtest_quotas (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:38:04.948ZDEBGtest_quotas: Starting Dendrite for switch0
152025-04-30T06:38:04.998ZDEBGtest_quotas (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:38:04.998ZDEBGtest_quotas: Starting mgd for switch0
172025-04-30T06:38:05.086ZDEBGtest_quotas: mgd port is 39922
182025-04-30T06:38:05.086ZDEBGtest_quotas (init_with_steps): Running step record_switch_dns
192025-04-30T06:38:05.086ZDEBGtest_quotas: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:38:05.086ZDEBGtest_quotas (init_with_steps): Running step start_internal_dns
212025-04-30T06:38:05.086ZINFOtest_quotas (store): opening sled database path = /var/tmp/omicron_tmp/.tmpbContx
222025-04-30T06:38:05.254ZINFOtest_quotas (store): pruning trees for generations newer than 0
232025-04-30T06:38:05.254ZINFOtest_quotas (store): pruning trees for generations older than 0 keep = 3
242025-04-30T06:38:05.255ZINFOtest_quotas (dns): DNS server bound to address kind = dns local_address = [::1]:34375
252025-04-30T06:38:05.255ZDEBGtest_quotas (http): registered endpoint kind = dns local_addr = [::1]:57119 method = GET path = /config versions = all
262025-04-30T06:38:05.255ZDEBGtest_quotas (http): registered endpoint kind = dns local_addr = [::1]:57119 method = PUT path = /config versions = all
272025-04-30T06:38:05.255ZINFOtest_quotas (http): listening kind = dns local_addr = [::1]:57119
282025-04-30T06:38:05.324ZDEBGtest_quotas (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:57119
292025-04-30T06:38:05.325ZDEBGtest_quotas (init_with_steps): Running step start_external_dns
302025-04-30T06:38:05.325ZINFOtest_quotas (store): opening sled database path = /var/tmp/omicron_tmp/.tmpMFRwiX
312025-04-30T06:38:05.342ZINFOtest_quotas (store): pruning trees for generations newer than 0
322025-04-30T06:38:05.342ZINFOtest_quotas (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:38:05.342ZINFOtest_quotas (dns): DNS server bound to address kind = dns local_address = [::1]:46209
342025-04-30T06:38:05.343ZDEBGtest_quotas (http): registered endpoint kind = dns local_addr = [::1]:36358 method = GET path = /config versions = all
352025-04-30T06:38:05.343ZDEBGtest_quotas (http): registered endpoint kind = dns local_addr = [::1]:36358 method = PUT path = /config versions = all
362025-04-30T06:38:05.343ZINFOtest_quotas (http): listening kind = dns local_addr = [::1]:36358
372025-04-30T06:38:05.344ZDEBGtest_quotas (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:36358
382025-04-30T06:38:05.344ZDEBGtest_quotas (init_with_steps): Running step start_nexus_internal
392025-04-30T06:38:05.344ZDEBGtest_quotas: Starting Nexus (internal API)
402025-04-30T06:38:05.345ZINFOtest_quotas: setting up nexus server
412025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = Action
422025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = AnyActor
432025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = AuthenticatedActor
442025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = BlueprintConfig
452025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = Database
462025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = DnsConfig
472025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = Fleet
482025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = Inventory
492025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = IpPoolList
502025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = ConsoleSessionList
512025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = DeviceAuthRequestList
522025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = SiloCertificateList
532025-04-30T06:38:05.348ZINFOtest_quotas (ServerContext): registering Oso class class = SiloIdentityProviderList
542025-04-30T06:38:05.349ZINFOtest_quotas (ServerContext): registering Oso class class = SiloUserList
552025-04-30T06:38:05.349ZINFOtest_quotas (ServerContext): registering Oso class class = TargetReleaseConfig
562025-04-30T06:38:05.349ZINFOtest_quotas (ServerContext): registering Oso class class = WebhookEventClassList
572025-04-30T06:38:05.349ZINFOtest_quotas (ServerContext): registering Oso class class = Project
582025-04-30T06:38:05.349ZINFOtest_quotas (ServerContext): registering Oso class class = Disk
592025-04-30T06:38:05.349ZINFOtest_quotas (ServerContext): registering Oso class class = Snapshot
602025-04-30T06:38:05.349ZINFOtest_quotas (ServerContext): registering Oso class class = ProjectImage
612025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = AffinityGroup
622025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = AntiAffinityGroup
632025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = Instance
642025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = IpPool
652025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = InstanceNetworkInterface
662025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = Vpc
672025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = VpcRouter
682025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = InternetGateway
692025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = InternetGatewayIpPool
702025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = InternetGatewayIpAddress
712025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = RouterRoute
722025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = VpcSubnet
732025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = FloatingIp
742025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = Image
752025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = SiloImage
762025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = AddressLot
772025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = Blueprint
782025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = LoopbackAddress
792025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = Certificate
802025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = ConsoleSession
812025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = DeviceAuthRequest
822025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = DeviceAccessToken
832025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = PhysicalDisk
842025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = Rack
852025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = RoleBuiltin
862025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = SshKey
872025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = Silo
882025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = SiloUser
892025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = SiloGroup
902025-04-30T06:38:05.350ZINFOtest_quotas (ServerContext): registering Oso class class = SupportBundle
912025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = IdentityProvider
922025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = SamlIdentityProvider
932025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = Sled
942025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = TufRepo
952025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = TufArtifact
962025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = WebhookEvent
972025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = WebhookReceiver
982025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = WebhookSecret
992025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = Zpool
1002025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = Service
1012025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): registering Oso class class = UserBuiltin
1022025-04-30T06:38:05.351ZINFOtest_quotas (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1032025-04-30T06:38:05.429ZINFOtest_quotas (ServerContext): Setting up resolver using DNS address: [::1]:34375
1042025-04-30T06:38:05.429ZINFOtest_quotas (DnsResolver): new DNS resolver addresses = [[::1]:34375]
1052025-04-30T06:38:05.429ZINFOtest_quotas (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:61168/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 61168,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1062025-04-30T06:38:05.431ZDEBGtest_quotas (ServerContext): registered USDT probes
1072025-04-30T06:38:05.531ZINFOtest_quotas (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1082025-04-30T06:38:05.712ZDEBGtest_quotas (clickhouse-client): registered USDT probes id = 303e7dcb-d152-4f00-8702-88476cfae85e
1092025-04-30T06:38:05.818ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /bgtasks versions = all
1102025-04-30T06:38:05.819ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /bgtasks/activate versions = all
1112025-04-30T06:38:05.819ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1122025-04-30T06:38:05.819ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /clickhouse/policy versions = all
1132025-04-30T06:38:05.819ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /clickhouse/policy versions = all
1142025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1152025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1162025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1172025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1182025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1192025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /demo-saga versions = all
1202025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1212025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /deployment/blueprints/all versions = all
1222025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1232025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1242025-04-30T06:38:05.820ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /deployment/blueprints/import versions = all
1252025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /deployment/blueprints/regenerate versions = all
1262025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /deployment/blueprints/target versions = all
1272025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /deployment/blueprints/target versions = all
1282025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = PUT path = /deployment/blueprints/target/enabled versions = all
1292025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1302025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = PUT path = /disks/{disk_id} versions = all
1312025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /experimental/v1/system/support-bundles versions = all
1322025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /experimental/v1/system/support-bundles versions = all
1332025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1342025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1352025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1362025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1372025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1382025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1392025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1402025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /instances/{instance_id}/migrate versions = all
1412025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /metrics/collectors versions = all
1422025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1432025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /metrics/producers versions = all
1442025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /mgs-updates versions = all
1452025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1462025-04-30T06:38:05.821ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /oximeter/read-policy versions = all
1472025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /oximeter/read-policy versions = all
1482025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /physical-disk/expunge versions = all
1492025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /probes/{sled} versions = all
1502025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1512025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /sagas versions = all
1522025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /sagas/{saga_id} versions = all
1532025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /sled-agents/{sled_id} versions = all
1542025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /sled-agents/{sled_id} versions = all
1552025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1562025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /sleds/add versions = all
1572025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /sleds/expunge versions = all
1582025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /sleds/uninitialized versions = all
1592025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = PUT path = /switch/{switch_id} versions = all
1602025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = GET path = /v1/ping versions = all
1612025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = PUT path = /vmms/{propolis_id} versions = all
1622025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): registered endpoint local_addr = 127.0.0.1:64801 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1632025-04-30T06:38:05.822ZINFOtest_quotas (dropshot_internal): listening local_addr = 127.0.0.1:64801
1642025-04-30T06:38:05.822ZDEBGtest_quotas (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:64801
1652025-04-30T06:38:05.822ZINFOtest_quotas (SEC): SEC running sec_id = 4678dd50-d1e0-4228-8533-c4090342ff58
1662025-04-30T06:38:05.824ZTRACtest_quotas (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1672025-04-30T06:38:05.824ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
1682025-04-30T06:38:05.827ZDEBGtest_quotas (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-04-30T06:38:05.828ZDEBGtest_quotas (DataLoader): attempting to create built-in users
1702025-04-30T06:38:05.828ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1712025-04-30T06:38:05.829ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
1722025-04-30T06:38:05.829ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-04-30T06:38:05.829ZINFOtest_quotas (MgsUpdateDriver): starting MgsUpdateDriver
1742025-04-30T06:38:05.831ZTRACtest_quotas (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: c7 a8 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:60617 req_id = 06d324ea-dcc8-4dac-a376-ee1b18ef831f
1752025-04-30T06:38:05.831ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51112,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60617 req_id = 06d324ea-dcc8-4dac-a376-ee1b18ef831f
1762025-04-30T06:38:05.833ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51112, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60617 req_id = 06d324ea-dcc8-4dac-a376-ee1b18ef831f
1772025-04-30T06:38:05.945ZINFOtest_quotas (DataLoader): created 5 built-in users
1782025-04-30T06:38:05.945ZTRACtest_quotas (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1792025-04-30T06:38:05.945ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
1802025-04-30T06:38:05.950ZDEBGtest_quotas (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1812025-04-30T06:38:05.950ZDEBGtest_quotas (DataLoader): attempting to create built-in roles
1822025-04-30T06:38:05.950ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1832025-04-30T06:38:05.950ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
1842025-04-30T06:38:05.953ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-04-30T06:38:05.986ZINFOtest_quotas (DataLoader): created 10 built-in roles
1862025-04-30T06:38:05.988ZTRACtest_quotas (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1872025-04-30T06:38:05.988ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
1882025-04-30T06:38:05.993ZDEBGtest_quotas (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-04-30T06:38:05.993ZDEBGtest_quotas (DataLoader): attempting to create built-in role assignments
1902025-04-30T06:38:05.993ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1912025-04-30T06:38:05.993ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
1922025-04-30T06:38:05.994ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-04-30T06:38:06.272ZINFOtest_quotas (DataLoader): created 4 built-in role assignments
1942025-04-30T06:38:06.273ZTRACtest_quotas (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1952025-04-30T06:38:06.273ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
1962025-04-30T06:38:06.274ZDEBGtest_quotas (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-04-30T06:38:06.275ZDEBGtest_quotas (DataLoader): attempting to create built-in silos
1982025-04-30T06:38:06.275ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1992025-04-30T06:38:06.275ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2002025-04-30T06:38:06.275ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-04-30T06:38:06.365ZINFOtest_quotas (DataLoader): created 2 built-in silos
2022025-04-30T06:38:06.365ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2032025-04-30T06:38:06.365ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2042025-04-30T06:38:06.365ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-04-30T06:38:06.387ZTRACtest_quotas (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2062025-04-30T06:38:06.387ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2072025-04-30T06:38:06.389ZDEBGtest_quotas (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2082025-04-30T06:38:06.389ZDEBGtest_quotas (DataLoader): attempting to create built-in projects
2092025-04-30T06:38:06.389ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2102025-04-30T06:38:06.389ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2112025-04-30T06:38:06.389ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2122025-04-30T06:38:06.394ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2132025-04-30T06:38:06.394ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2142025-04-30T06:38:06.394ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:38:06.394ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:38:06.395ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:38:06.460ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2182025-04-30T06:38:06.460ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:38:06.460ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:38:06.461ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:38:06.464ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2222025-04-30T06:38:06.467ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2232025-04-30T06:38:06.467ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2242025-04-30T06:38:06.467ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2252025-04-30T06:38:06.467ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2262025-04-30T06:38:06.467ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2272025-04-30T06:38:06.467ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-04-30T06:38:06.470ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2292025-04-30T06:38:06.470ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2302025-04-30T06:38:06.470ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:38:06.470ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-04-30T06:38:06.473ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2332025-04-30T06:38:06.475ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2342025-04-30T06:38:06.476ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:38:06.476ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:38:06.476ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:38:06.495ZINFOtest_quotas (DataLoader): created built-in services project
2382025-04-30T06:38:06.495ZTRACtest_quotas (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:38:06.495ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:38:06.497ZDEBGtest_quotas (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:38:06.497ZDEBGtest_quotas (DataLoader): attempting to create built-in VPCs
2422025-04-30T06:38:06.497ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2432025-04-30T06:38:06.497ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2442025-04-30T06:38:06.498ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-04-30T06:38:06.502ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2462025-04-30T06:38:06.502ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2472025-04-30T06:38:06.503ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-04-30T06:38:06.510ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2492025-04-30T06:38:06.510ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2502025-04-30T06:38:06.510ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2512025-04-30T06:38:06.510ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2522025-04-30T06:38:06.511ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2532025-04-30T06:38:06.513ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2542025-04-30T06:38:06.513ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:38:06.513ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:38:06.515ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:38:06.516ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2582025-04-30T06:38:06.516ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2592025-04-30T06:38:06.516ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2602025-04-30T06:38:06.517ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-04-30T06:38:06.519ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2622025-04-30T06:38:06.526ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2632025-04-30T06:38:06.526ZTRACtest_quotas (DataLoader): successful lookup of siloed resource "Project" using built-in user
2642025-04-30T06:38:06.526ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2652025-04-30T06:38:06.526ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2662025-04-30T06:38:06.526ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2672025-04-30T06:38:06.526ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2682025-04-30T06:38:06.527ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2692025-04-30T06:38:06.529ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2702025-04-30T06:38:06.529ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2712025-04-30T06:38:06.529ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2722025-04-30T06:38:06.529ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-04-30T06:38:06.532ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2742025-04-30T06:38:06.532ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2752025-04-30T06:38:06.532ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2762025-04-30T06:38:06.533ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2772025-04-30T06:38:06.535ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2782025-04-30T06:38:06.541ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2792025-04-30T06:38:06.541ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:38:06.541ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:38:06.541ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:38:06.557ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2832025-04-30T06:38:06.557ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2842025-04-30T06:38:06.557ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2852025-04-30T06:38:06.557ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2862025-04-30T06:38:06.559ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2872025-04-30T06:38:06.561ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2882025-04-30T06:38:06.561ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:38:06.561ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:38:06.562ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:38:06.564ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2922025-04-30T06:38:06.564ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2932025-04-30T06:38:06.564ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2942025-04-30T06:38:06.564ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-04-30T06:38:06.569ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2962025-04-30T06:38:06.578ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2972025-04-30T06:38:06.578ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:38:06.578ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:38:06.579ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:38:06.586ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3012025-04-30T06:38:06.586ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3022025-04-30T06:38:06.589ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-04-30T06:38:06.597ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3042025-04-30T06:38:06.598ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:38:06.598ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3062025-04-30T06:38:06.604ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3072025-04-30T06:38:06.604ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3082025-04-30T06:38:06.604ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3092025-04-30T06:38:06.604ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3102025-04-30T06:38:06.604ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-04-30T06:38:06.608ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3122025-04-30T06:38:06.609ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:38:06.609ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:38:06.609ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:38:06.611ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3162025-04-30T06:38:06.611ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3172025-04-30T06:38:06.611ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3182025-04-30T06:38:06.612ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-04-30T06:38:06.617ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3202025-04-30T06:38:06.623ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3212025-04-30T06:38:06.623ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:38:06.623ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:38:06.623ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:38:06.714ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3252025-04-30T06:38:06.714ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3262025-04-30T06:38:06.714ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3272025-04-30T06:38:06.714ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3282025-04-30T06:38:06.715ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-04-30T06:38:06.722ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3302025-04-30T06:38:06.722ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:38:06.722ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:38:06.723ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:38:06.730ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3342025-04-30T06:38:06.730ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3352025-04-30T06:38:06.730ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3362025-04-30T06:38:06.731ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3372025-04-30T06:38:06.734ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3382025-04-30T06:38:06.756ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3392025-04-30T06:38:06.756ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:38:06.756ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:38:06.757ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:38:06.852ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3432025-04-30T06:38:06.853ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3442025-04-30T06:38:06.853ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3452025-04-30T06:38:06.853ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3462025-04-30T06:38:06.854ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-04-30T06:38:06.856ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3482025-04-30T06:38:06.856ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:38:06.856ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:38:06.858ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:38:06.863ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3522025-04-30T06:38:06.864ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3532025-04-30T06:38:06.864ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3542025-04-30T06:38:06.864ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-04-30T06:38:06.874ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3562025-04-30T06:38:06.882ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3572025-04-30T06:38:06.882ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:38:06.883ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:38:06.883ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:38:06.891ZDEBGtest_quotas (DataLoader): attempting to create built-in VPC firewall rules
3612025-04-30T06:38:06.891ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:38:06.891ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:38:06.892ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:38:06.898ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3652025-04-30T06:38:06.898ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3662025-04-30T06:38:06.899ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3672025-04-30T06:38:06.904ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3682025-04-30T06:38:06.904ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3692025-04-30T06:38:06.905ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-04-30T06:38:06.912ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3712025-04-30T06:38:06.912ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3722025-04-30T06:38:06.912ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3732025-04-30T06:38:06.912ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3742025-04-30T06:38:06.913ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-04-30T06:38:07.014ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3762025-04-30T06:38:07.014ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:38:07.014ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:38:07.014ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:38:07.026ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3802025-04-30T06:38:07.026ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3812025-04-30T06:38:07.026ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3822025-04-30T06:38:07.026ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-04-30T06:38:07.031ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3842025-04-30T06:38:07.040ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3852025-04-30T06:38:07.040ZTRACtest_quotas (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3862025-04-30T06:38:07.040ZTRACtest_quotas (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3872025-04-30T06:38:07.040ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3882025-04-30T06:38:07.040ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3892025-04-30T06:38:07.040ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3902025-04-30T06:38:07.041ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3912025-04-30T06:38:07.049ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3922025-04-30T06:38:07.049ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3932025-04-30T06:38:07.049ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3942025-04-30T06:38:07.050ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3952025-04-30T06:38:07.053ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3962025-04-30T06:38:07.053ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3972025-04-30T06:38:07.053ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
3982025-04-30T06:38:07.054ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3992025-04-30T06:38:07.056ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4002025-04-30T06:38:07.068ZDEBGtest_quotas (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4012025-04-30T06:38:07.068ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:38:07.068ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:38:07.069ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:38:07.080ZTRACtest_quotas (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4052025-04-30T06:38:07.080ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4062025-04-30T06:38:07.080ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4072025-04-30T06:38:07.080ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4082025-04-30T06:38:07.081ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4092025-04-30T06:38:07.083ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4102025-04-30T06:38:07.083ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:38:07.083ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:38:07.084ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:38:07.090ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4142025-04-30T06:38:07.090ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4152025-04-30T06:38:07.090ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4162025-04-30T06:38:07.090ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4172025-04-30T06:38:07.094ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4182025-04-30T06:38:07.107ZDEBGtest_quotas (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4192025-04-30T06:38:07.107ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:38:07.107ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:38:07.108ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:38:07.235ZINFOtest_quotas (DataLoader): created built-in VPC firewall rules
4232025-04-30T06:38:07.236ZDEBGtest_quotas (DataLoader): attempting to create built-in VPC Subnets
4242025-04-30T06:38:07.236ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4252025-04-30T06:38:07.236ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4262025-04-30T06:38:07.236ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4272025-04-30T06:38:07.242ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4282025-04-30T06:38:07.242ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4292025-04-30T06:38:07.243ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4302025-04-30T06:38:07.246ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4312025-04-30T06:38:07.246ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4322025-04-30T06:38:07.246ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4332025-04-30T06:38:07.260ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4342025-04-30T06:38:07.260ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4352025-04-30T06:38:07.260ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4362025-04-30T06:38:07.260ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4372025-04-30T06:38:07.261ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4382025-04-30T06:38:07.264ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4392025-04-30T06:38:07.264ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:38:07.264ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:38:07.265ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:38:07.267ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4432025-04-30T06:38:07.267ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4442025-04-30T06:38:07.267ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4452025-04-30T06:38:07.268ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4462025-04-30T06:38:07.271ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4472025-04-30T06:38:07.276ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4482025-04-30T06:38:07.276ZTRACtest_quotas (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4492025-04-30T06:38:07.276ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4502025-04-30T06:38:07.276ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4512025-04-30T06:38:07.277ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4522025-04-30T06:38:07.283ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4532025-04-30T06:38:07.283ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4542025-04-30T06:38:07.283ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4552025-04-30T06:38:07.283ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:38:07.284ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4572025-04-30T06:38:07.286ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4582025-04-30T06:38:07.286ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:38:07.286ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:38:07.287ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:38:07.289ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4622025-04-30T06:38:07.289ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4632025-04-30T06:38:07.289ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4642025-04-30T06:38:07.289ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4652025-04-30T06:38:07.292ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4662025-04-30T06:38:07.298ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4672025-04-30T06:38:07.305ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4682025-04-30T06:38:07.306ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4692025-04-30T06:38:07.306ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4702025-04-30T06:38:07.306ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4712025-04-30T06:38:07.306ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4722025-04-30T06:38:07.308ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4732025-04-30T06:38:07.308ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4742025-04-30T06:38:07.308ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:38:07.309ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4762025-04-30T06:38:07.311ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4772025-04-30T06:38:07.311ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4782025-04-30T06:38:07.311ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4792025-04-30T06:38:07.312ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4802025-04-30T06:38:07.314ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4812025-04-30T06:38:07.321ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4822025-04-30T06:38:07.321ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:38:07.321ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:38:07.321ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:38:07.380ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4862025-04-30T06:38:07.380ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4872025-04-30T06:38:07.381ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4882025-04-30T06:38:07.386ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4892025-04-30T06:38:07.386ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4902025-04-30T06:38:07.386ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4912025-04-30T06:38:07.386ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4922025-04-30T06:38:07.387ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4932025-04-30T06:38:07.389ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4942025-04-30T06:38:07.389ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:38:07.389ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:38:07.390ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:38:07.392ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4982025-04-30T06:38:07.392ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4992025-04-30T06:38:07.392ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5002025-04-30T06:38:07.392ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5012025-04-30T06:38:07.394ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5022025-04-30T06:38:07.400ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5032025-04-30T06:38:07.405ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5042025-04-30T06:38:07.405ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5052025-04-30T06:38:07.405ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5062025-04-30T06:38:07.405ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5072025-04-30T06:38:07.406ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5082025-04-30T06:38:07.408ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5092025-04-30T06:38:07.408ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5102025-04-30T06:38:07.408ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:38:07.408ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5122025-04-30T06:38:07.410ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5132025-04-30T06:38:07.410ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5142025-04-30T06:38:07.410ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5152025-04-30T06:38:07.411ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5162025-04-30T06:38:07.415ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5172025-04-30T06:38:07.421ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5182025-04-30T06:38:07.421ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:38:07.421ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:38:07.422ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:38:07.440ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5222025-04-30T06:38:07.441ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5232025-04-30T06:38:07.442ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5242025-04-30T06:38:07.444ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5252025-04-30T06:38:07.444ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5262025-04-30T06:38:07.444ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5272025-04-30T06:38:07.444ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5282025-04-30T06:38:07.444ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5292025-04-30T06:38:07.446ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5302025-04-30T06:38:07.446ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:38:07.446ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:38:07.447ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:38:07.449ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5342025-04-30T06:38:07.449ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5352025-04-30T06:38:07.449ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5362025-04-30T06:38:07.450ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5372025-04-30T06:38:07.452ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5382025-04-30T06:38:07.457ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5392025-04-30T06:38:07.465ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5402025-04-30T06:38:07.465ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5412025-04-30T06:38:07.465ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5422025-04-30T06:38:07.465ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5432025-04-30T06:38:07.467ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5442025-04-30T06:38:07.469ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5452025-04-30T06:38:07.469ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5462025-04-30T06:38:07.469ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:38:07.469ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5482025-04-30T06:38:07.474ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5492025-04-30T06:38:07.474ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5502025-04-30T06:38:07.474ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5512025-04-30T06:38:07.520ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5522025-04-30T06:38:07.526ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5532025-04-30T06:38:07.534ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5542025-04-30T06:38:07.534ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:38:07.534ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:38:07.534ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:38:07.576ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5582025-04-30T06:38:07.576ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5592025-04-30T06:38:07.577ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5602025-04-30T06:38:07.582ZINFOtest_quotas (DataLoader): created built-in services vpc subnets
5612025-04-30T06:38:07.582ZINFOtest_quotas (DataLoader): created built-in services vpc
5622025-04-30T06:38:07.582ZTRACtest_quotas (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5632025-04-30T06:38:07.582ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5642025-04-30T06:38:07.589ZDEBGtest_quotas (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5652025-04-30T06:38:07.589ZDEBGtest_quotas (DataLoader): attempting to create silo users
5662025-04-30T06:38:07.589ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:38:07.589ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:38:07.590ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:38:07.604ZINFOtest_quotas (DataLoader): created 2 silo users
5702025-04-30T06:38:07.604ZDEBGtest_quotas (DataLoader): attempting to create silo user role assignments
5712025-04-30T06:38:07.605ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:38:07.605ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:38:07.607ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:38:07.619ZINFOtest_quotas (DataLoader): created 2 silo user role assignments
5752025-04-30T06:38:07.619ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:38:07.619ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:38:07.620ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:38:07.628ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5792025-04-30T06:38:07.628ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5802025-04-30T06:38:07.628ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5812025-04-30T06:38:07.645ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5822025-04-30T06:38:07.645ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5832025-04-30T06:38:07.645ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5842025-04-30T06:38:07.646ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5852025-04-30T06:38:07.647ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5862025-04-30T06:38:07.657ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5872025-04-30T06:38:07.661ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5882025-04-30T06:38:07.661ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5892025-04-30T06:38:07.661ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5902025-04-30T06:38:07.661ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5912025-04-30T06:38:07.671ZTRACtest_quotas (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5922025-04-30T06:38:07.671ZTRACtest_quotas (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5932025-04-30T06:38:07.671ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5942025-04-30T06:38:07.671ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5952025-04-30T06:38:07.672ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5962025-04-30T06:38:07.695ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
5972025-04-30T06:38:07.697ZDEBGtest_quotas (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5982025-04-30T06:38:07.697ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5992025-04-30T06:38:07.697ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
6002025-04-30T06:38:07.698ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6012025-04-30T06:38:07.800ZTRACtest_quotas (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6022025-04-30T06:38:07.800ZDEBGtest_quotas (DataLoader): roles roles = RoleSet { roles: {} }
6032025-04-30T06:38:07.801ZDEBGtest_quotas (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6042025-04-30T06:38:07.809ZINFOtest_quotas (nexus): populate complete
6052025-04-30T06:38:07.809ZINFOtest_quotas (nexus): activating background tasks
6062025-04-30T06:38:07.818ZDEBGtest_quotas (init_with_steps): Running step start_sled1
6072025-04-30T06:38:07.818ZINFOtest_quotas (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6082025-04-30T06:38:07.877ZINFOtest_quotas (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6092025-04-30T06:38:07.878ZINFOtest_quotas (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6102025-04-30T06:38:07.920ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6112025-04-30T06:38:07.920ZDEBGtest_quotas (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6122025-04-30T06:38:07.920ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6132025-04-30T06:38:07.920ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142025-04-30T06:38:07.921ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6152025-04-30T06:38:07.923ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6162025-04-30T06:38:07.923ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6172025-04-30T06:38:07.924ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6182025-04-30T06:38:07.924ZWARNtest_quotas (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6192025-04-30T06:38:07.924ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 271.952\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6202025-04-30T06:38:07.924ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6212025-04-30T06:38:07.924ZDEBGtest_quotas (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6222025-04-30T06:38:07.924ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
6232025-04-30T06:38:07.925ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6242025-04-30T06:38:07.925ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6252025-04-30T06:38:07.925ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
6262025-04-30T06:38:07.925ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
6272025-04-30T06:38:07.926ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
6282025-04-30T06:38:07.926ZWARNtest_quotas (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
6292025-04-30T06:38:07.926ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 52.224\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6302025-04-30T06:38:07.926ZDEBGtest_quotas (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
6312025-04-30T06:38:07.926ZINFOtest_quotas (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:28:07.926584196 UTC
6322025-04-30T06:38:07.926ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
6332025-04-30T06:38:07.927ZDEBGtest_quotas (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
6342025-04-30T06:38:07.927ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
6352025-04-30T06:38:07.927ZDEBGtest_quotas (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
6362025-04-30T06:38:07.927ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
6372025-04-30T06:38:07.927ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6382025-04-30T06:38:07.928ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
6392025-04-30T06:38:07.928ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6402025-04-30T06:38:07.931ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6412025-04-30T06:38:07.931ZDEBGtest_quotas (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
6422025-04-30T06:38:07.931ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
6432025-04-30T06:38:07.931ZDEBGtest_quotas (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6442025-04-30T06:38:07.931ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6452025-04-30T06:38:07.931ZDEBGtest_quotas (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
6462025-04-30T06:38:07.932ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
6472025-04-30T06:38:07.932ZDEBGtest_quotas (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6482025-04-30T06:38:07.932ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6492025-04-30T06:38:07.932ZDEBGtest_quotas (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
6502025-04-30T06:38:07.932ZDEBGtest_quotas (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
6512025-04-30T06:38:07.932ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
6522025-04-30T06:38:07.932ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6532025-04-30T06:38:07.932ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
6542025-04-30T06:38:07.932ZDEBGtest_quotas (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6552025-04-30T06:38:07.933ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6562025-04-30T06:38:07.933ZDEBGtest_quotas (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
6572025-04-30T06:38:07.933ZWARNtest_quotas (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
6582025-04-30T06:38:07.933ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 56.382\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6592025-04-30T06:38:07.933ZDEBGtest_quotas (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
6602025-04-30T06:38:07.933ZWARNtest_quotas (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
6612025-04-30T06:38:07.933ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 20.851\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6622025-04-30T06:38:07.933ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
6632025-04-30T06:38:07.933ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
6642025-04-30T06:38:07.933ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6652025-04-30T06:38:07.933ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
6662025-04-30T06:38:07.933ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
6672025-04-30T06:38:07.933ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
6682025-04-30T06:38:07.933ZDEBGtest_quotas (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
6692025-04-30T06:38:07.933ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
6702025-04-30T06:38:07.933ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6712025-04-30T06:38:07.933ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
6722025-04-30T06:38:07.934ZDEBGtest_quotas (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6732025-04-30T06:38:07.934ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6742025-04-30T06:38:07.934ZDEBGtest_quotas (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
6752025-04-30T06:38:07.934ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 22.795\u{b5}s iteration = 1 status = {"error":"task disabled"}
6762025-04-30T06:38:07.934ZDEBGtest_quotas (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
6772025-04-30T06:38:07.934ZWARNtest_quotas (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
6782025-04-30T06:38:07.934ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 13.868\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6792025-04-30T06:38:07.934ZDEBGtest_quotas (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
6802025-04-30T06:38:07.934ZINFOtest_quotas (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
6812025-04-30T06:38:07.934ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 83.275\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6822025-04-30T06:38:07.934ZDEBGtest_quotas (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
6832025-04-30T06:38:07.934ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
6842025-04-30T06:38:07.934ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6852025-04-30T06:38:07.934ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
6862025-04-30T06:38:07.934ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6872025-04-30T06:38:07.935ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6882025-04-30T06:38:07.935ZDEBGtest_quotas (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
6892025-04-30T06:38:07.935ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
6902025-04-30T06:38:07.935ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6912025-04-30T06:38:07.935ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
6922025-04-30T06:38:07.935ZDEBGtest_quotas (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6932025-04-30T06:38:07.936ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6942025-04-30T06:38:07.938ZDEBGtest_quotas (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
6952025-04-30T06:38:07.943ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
6962025-04-30T06:38:07.943ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6972025-04-30T06:38:07.944ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6982025-04-30T06:38:07.944ZDEBGtest_quotas (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
6992025-04-30T06:38:07.944ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7002025-04-30T06:38:07.944ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7012025-04-30T06:38:07.944ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7022025-04-30T06:38:07.944ZDEBGtest_quotas (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7032025-04-30T06:38:07.944ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7042025-04-30T06:38:07.944ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7052025-04-30T06:38:07.945ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7062025-04-30T06:38:07.945ZDEBGtest_quotas (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7072025-04-30T06:38:07.945ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7082025-04-30T06:38:07.945ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7092025-04-30T06:38:07.945ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7102025-04-30T06:38:07.945ZDEBGtest_quotas (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7112025-04-30T06:38:07.946ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7122025-04-30T06:38:07.949ZDEBGtest_quotas (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7132025-04-30T06:38:07.949ZINFOtest_quotas (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7142025-04-30T06:38:07.950ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = instance_updater elapsed = 460.979\u{b5}s iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7152025-04-30T06:38:07.950ZDEBGtest_quotas (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7162025-04-30T06:38:07.950ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7172025-04-30T06:38:07.950ZDEBGtest_quotas (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7182025-04-30T06:38:07.952ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7192025-04-30T06:38:07.952ZDEBGtest_quotas (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7202025-04-30T06:38:07.952ZDEBGtest_quotas (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7212025-04-30T06:38:07.952ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
7222025-04-30T06:38:07.952ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7232025-04-30T06:38:07.953ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7242025-04-30T06:38:07.953ZDEBGtest_quotas (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
7252025-04-30T06:38:07.953ZINFOtest_quotas (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
7262025-04-30T06:38:07.953ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
7272025-04-30T06:38:07.953ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7282025-04-30T06:38:07.953ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
7292025-04-30T06:38:07.953ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7302025-04-30T06:38:07.954ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7312025-04-30T06:38:07.954ZDEBGtest_quotas (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
7322025-04-30T06:38:07.956ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
7332025-04-30T06:38:07.956ZDEBGtest_quotas (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
7342025-04-30T06:38:07.956ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
7352025-04-30T06:38:07.956ZDEBGtest_quotas (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
7362025-04-30T06:38:07.957ZDEBGtest_quotas (SagaRecoverer): listing candidate sagas for recovery
7372025-04-30T06:38:07.957ZTRACtest_quotas (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
7382025-04-30T06:38:07.957ZDEBGtest_quotas (SagaRecoverer): roles roles = RoleSet { roles: {} }
7392025-04-30T06:38:07.957ZDEBGtest_quotas (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
7402025-04-30T06:38:07.957ZDEBGtest_quotas (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
7412025-04-30T06:38:07.957ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
7422025-04-30T06:38:07.957ZDEBGtest_quotas (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
7432025-04-30T06:38:07.957ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
7442025-04-30T06:38:07.957ZDEBGtest_quotas (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
7452025-04-30T06:38:07.957ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
7462025-04-30T06:38:07.957ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7472025-04-30T06:38:07.958ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
7482025-04-30T06:38:07.958ZDEBGtest_quotas (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
7492025-04-30T06:38:07.958ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
7502025-04-30T06:38:07.958ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
7512025-04-30T06:38:07.958ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
7522025-04-30T06:38:07.958ZDEBGtest_quotas (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
7532025-04-30T06:38:07.958ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
7542025-04-30T06:38:07.958ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7552025-04-30T06:38:07.958ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7562025-04-30T06:38:07.958ZDEBGtest_quotas (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
7572025-04-30T06:38:07.958ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
7582025-04-30T06:38:07.959ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
7592025-04-30T06:38:07.960ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
7602025-04-30T06:38:07.960ZDEBGtest_quotas (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
7612025-04-30T06:38:07.960ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
7622025-04-30T06:38:07.960ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7632025-04-30T06:38:07.960ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
7642025-04-30T06:38:07.960ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7652025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7662025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
7672025-04-30T06:38:07.969ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
7682025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
7692025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
7702025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
7712025-04-30T06:38:07.969ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
7722025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
7732025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
7742025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
7752025-04-30T06:38:07.969ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
7762025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
7772025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
7782025-04-30T06:38:07.969ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
7792025-04-30T06:38:07.969ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7802025-04-30T06:38:07.969ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
7812025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7822025-04-30T06:38:07.969ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7832025-04-30T06:38:07.969ZINFOtest_quotas (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp3gqMPc/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7842025-04-30T06:38:07.969ZDEBGtest_quotas (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp3gqMPc/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7852025-04-30T06:38:07.969ZINFOtest_quotas (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpsOhJx1/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7862025-04-30T06:38:07.969ZDEBGtest_quotas (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpsOhJx1/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7872025-04-30T06:38:08.067ZDEBGtest_quotas (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:44150 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7882025-04-30T06:38:08.067ZINFOtest_quotas (dropshot (Repo Depot)): listening local_addr = [::1]:44150 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7892025-04-30T06:38:08.070ZDEBGtest_quotas (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:44150 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7902025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7912025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7922025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7932025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7942025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7952025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7962025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7972025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7982025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7992025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8002025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8012025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8022025-04-30T06:38:08.080ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8032025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8042025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8052025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8062025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8072025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8082025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8092025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8102025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8112025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8122025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8132025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8142025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8152025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8162025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8172025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8182025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8192025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8202025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8212025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8222025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8232025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8242025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8252025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8262025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8272025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8282025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8292025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8302025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8312025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8322025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8332025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8342025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8352025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8362025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8372025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8382025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8392025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8402025-04-30T06:38:08.081ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8412025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8422025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8432025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8442025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8452025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8462025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8472025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8482025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8492025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8502025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8512025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8522025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8532025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8542025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8552025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8562025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8572025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8582025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:55655 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8592025-04-30T06:38:08.082ZINFOtest_quotas (dropshot): listening local_addr = [::1]:55655 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8602025-04-30T06:38:08.082ZDEBGtest_quotas (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:55655 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8612025-04-30T06:38:08.082ZDEBGtest_quotas (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8622025-04-30T06:38:08.084ZDEBGtest_quotas (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:64801/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8632025-04-30T06:38:08.085ZTRACtest_quotas (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: c7 ad 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:65300 req_id = eb06d0a3-0dda-47e7-92f3-6f871549fee4
8642025-04-30T06:38:08.085ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51117,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65300 req_id = eb06d0a3-0dda-47e7-92f3-6f871549fee4
8652025-04-30T06:38:08.086ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51117, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65300 req_id = eb06d0a3-0dda-47e7-92f3-6f871549fee4
8662025-04-30T06:38:08.088ZTRACtest_quotas (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 0d 8d 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65090 req_id = ba47bcad-7d4b-40dc-906a-9e9bf1ebd391
8672025-04-30T06:38:08.088ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3469,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65090 req_id = ba47bcad-7d4b-40dc-906a-9e9bf1ebd391
8682025-04-30T06:38:08.089ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3469, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65090 req_id = ba47bcad-7d4b-40dc-906a-9e9bf1ebd391
8692025-04-30T06:38:08.089ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8702025-04-30T06:38:08.089ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8712025-04-30T06:38:08.089ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8722025-04-30T06:38:08.089ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8732025-04-30T06:38:08.090ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8742025-04-30T06:38:08.091ZINFOtest_quotas (dropshot_internal): accepted connection local_addr = 127.0.0.1:64801 remote_addr = 127.0.0.1:60161
8752025-04-30T06:38:08.091ZWARNtest_quotas (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8762025-04-30T06:38:08.091ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 168.116972ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8772025-04-30T06:38:08.092ZWARNtest_quotas (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8782025-04-30T06:38:08.092ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 166.350884ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8792025-04-30T06:38:08.106ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:60161 req_id = f35e448e-bb40-444c-ac9b-8067f9ea9af5 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8802025-04-30T06:38:08.109ZINFOtest_quotas (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8812025-04-30T06:38:08.112ZINFOtest_quotas (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:28:07.926584196 UTC pruned = {}
8822025-04-30T06:38:08.112ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 185.350128ms iteration = 1 status = {"expiration":"2025-04-30T06:28:07.926584196Z","pruned":[]}
8832025-04-30T06:38:08.124ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8842025-04-30T06:38:08.125ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8852025-04-30T06:38:08.134ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: fa aa 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52129 req_id = 15a5b5d5-ca0d-4e55-adf2-5cba5938a669
8862025-04-30T06:38:08.134ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64170,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52129 req_id = 15a5b5d5-ca0d-4e55-adf2-5cba5938a669
8872025-04-30T06:38:08.135ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64170, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52129 req_id = 15a5b5d5-ca0d-4e55-adf2-5cba5938a669
8882025-04-30T06:38:08.135ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 7e 5b 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ~[..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64832 req_id = d1f1843b-1bc6-4dbc-93db-1a958c2235b1
8892025-04-30T06:38:08.135ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32347,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64832 req_id = d1f1843b-1bc6-4dbc-93db-1a958c2235b1
8902025-04-30T06:38:08.135ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32347, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64832 req_id = d1f1843b-1bc6-4dbc-93db-1a958c2235b1
8912025-04-30T06:38:08.137ZERROtest_quotas (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8922025-04-30T06:38:08.137ZERROtest_quotas (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8932025-04-30T06:38:08.137ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 206.328286ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8942025-04-30T06:38:08.137ZERROtest_quotas (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8952025-04-30T06:38:08.137ZERROtest_quotas (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8962025-04-30T06:38:08.137ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 205.962908ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8972025-04-30T06:38:08.182ZDEBGtest_quotas (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8982025-04-30T06:38:08.202ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8992025-04-30T06:38:08.203ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
9002025-04-30T06:38:08.203ZDEBGtest_quotas (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9012025-04-30T06:38:08.203ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9022025-04-30T06:38:08.234ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 301.523566ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
9032025-04-30T06:38:08.275ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9042025-04-30T06:38:08.288ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9052025-04-30T06:38:08.288ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9062025-04-30T06:38:08.288ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9072025-04-30T06:38:08.289ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9082025-04-30T06:38:08.290ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9092025-04-30T06:38:08.294ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9102025-04-30T06:38:08.294ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
9112025-04-30T06:38:08.294ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9122025-04-30T06:38:08.295ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9132025-04-30T06:38:08.446ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9142025-04-30T06:38:08.446ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9152025-04-30T06:38:08.448ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9162025-04-30T06:38:08.546ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9172025-04-30T06:38:08.564ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9182025-04-30T06:38:08.564ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9192025-04-30T06:38:08.564ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9202025-04-30T06:38:08.565ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9212025-04-30T06:38:08.572ZDEBGtest_quotas (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9222025-04-30T06:38:08.576ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9232025-04-30T06:38:08.576ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9242025-04-30T06:38:08.576ZDEBGtest_quotas (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9252025-04-30T06:38:08.577ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9262025-04-30T06:38:08.577ZDEBGtest_quotas (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9272025-04-30T06:38:08.581ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9282025-04-30T06:38:08.581ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9292025-04-30T06:38:08.581ZDEBGtest_quotas (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9302025-04-30T06:38:08.581ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9312025-04-30T06:38:08.665ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9322025-04-30T06:38:08.670ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9332025-04-30T06:38:08.670ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9342025-04-30T06:38:08.670ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9352025-04-30T06:38:08.670ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9362025-04-30T06:38:08.670ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9372025-04-30T06:38:08.670ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9382025-04-30T06:38:08.670ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 714.834241ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9392025-04-30T06:38:08.776ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9402025-04-30T06:38:08.776ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9412025-04-30T06:38:08.777ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9422025-04-30T06:38:08.779ZDEBGtest_quotas (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9432025-04-30T06:38:08.786ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9442025-04-30T06:38:08.790ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9452025-04-30T06:38:08.790ZDEBGtest_quotas (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9462025-04-30T06:38:08.793ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9472025-04-30T06:38:08.810ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9482025-04-30T06:38:08.810ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9492025-04-30T06:38:08.811ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9502025-04-30T06:38:08.951ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 994.23617ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
9512025-04-30T06:38:08.951ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9522025-04-30T06:38:08.951ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9532025-04-30T06:38:08.952ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9542025-04-30T06:38:08.975ZTRACtest_quotas (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9552025-04-30T06:38:08.975ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9562025-04-30T06:38:08.975ZDEBGtest_quotas (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9572025-04-30T06:38:08.976ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9582025-04-30T06:38:09.068ZDEBGtest_quotas (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9592025-04-30T06:38:09.068ZINFOtest_quotas (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9602025-04-30T06:38:09.069ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 1.114858281s iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9612025-04-30T06:38:09.069ZINFOtest_quotas (SagaRecoverer): listed in-progress sagas count = 0
9622025-04-30T06:38:09.069ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 1.113901265s iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9632025-04-30T06:38:09.080ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 1.118762661s iteration = 1 status = {"errors":[],"requests_created_ok":[]}
9642025-04-30T06:38:09.086ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 1.127429058s iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
9652025-04-30T06:38:09.088ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9662025-04-30T06:38:09.092ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9672025-04-30T06:38:09.092ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9682025-04-30T06:38:09.092ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9692025-04-30T06:38:09.092ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9702025-04-30T06:38:09.092ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9712025-04-30T06:38:09.093ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9722025-04-30T06:38:09.099ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9732025-04-30T06:38:09.099ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9742025-04-30T06:38:09.099ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9752025-04-30T06:38:09.102ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 1.136599199s iteration = 1 status = {"by_rx":{},"error":null}
9762025-04-30T06:38:09.104ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9772025-04-30T06:38:09.111ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9782025-04-30T06:38:09.112ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
9792025-04-30T06:38:09.112ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9802025-04-30T06:38:09.113ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9812025-04-30T06:38:09.210ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9822025-04-30T06:38:09.210ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9832025-04-30T06:38:09.210ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9842025-04-30T06:38:09.210ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9852025-04-30T06:38:09.211ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9862025-04-30T06:38:09.221ZWARNtest_quotas (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
9872025-04-30T06:38:09.221ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 1.289300367s iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9882025-04-30T06:38:09.223ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9892025-04-30T06:38:09.223ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9902025-04-30T06:38:09.224ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9912025-04-30T06:38:09.224ZWARNtest_quotas (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
9922025-04-30T06:38:09.224ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 1.289967556s iteration = 1 status = {"error":"inventory collection is None"}
9932025-04-30T06:38:09.228ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 1.29221283s iteration = 1 status = {}
9942025-04-30T06:38:09.228ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9952025-04-30T06:38:09.231ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9962025-04-30T06:38:09.231ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9972025-04-30T06:38:09.231ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9982025-04-30T06:38:09.232ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9992025-04-30T06:38:09.235ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
10002025-04-30T06:38:09.235ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
10012025-04-30T06:38:09.235ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c8 fc 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65235 req_id = 4f0418b4-48f4-4922-bd67-9277940a2c5a
10022025-04-30T06:38:09.235ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51452,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65235 req_id = 4f0418b4-48f4-4922-bd67-9277940a2c5a
10032025-04-30T06:38:09.236ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51452, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65235 req_id = 4f0418b4-48f4-4922-bd67-9277940a2c5a
10042025-04-30T06:38:09.236ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
10052025-04-30T06:38:09.236ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.303396803s iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
10062025-04-30T06:38:09.370ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10072025-04-30T06:38:09.374ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10082025-04-30T06:38:09.374ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10092025-04-30T06:38:09.375ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10102025-04-30T06:38:09.375ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10112025-04-30T06:38:09.400ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10122025-04-30T06:38:09.406ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10132025-04-30T06:38:09.502ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10142025-04-30T06:38:09.502ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10152025-04-30T06:38:09.502ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10162025-04-30T06:38:09.502ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10172025-04-30T06:38:09.502ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10182025-04-30T06:38:09.505ZTRACtest_quotas (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10192025-04-30T06:38:09.506ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10202025-04-30T06:38:09.506ZDEBGtest_quotas (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10212025-04-30T06:38:09.507ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10222025-04-30T06:38:09.510ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 1.566246965s iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10232025-04-30T06:38:09.512ZINFOtest_quotas (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10242025-04-30T06:38:09.512ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 1.566825187s iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10252025-04-30T06:38:09.515ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
10262025-04-30T06:38:09.515ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10272025-04-30T06:38:09.515ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10282025-04-30T06:38:09.515ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10292025-04-30T06:38:09.531ZTRACtest_quotas (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
10302025-04-30T06:38:09.531ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 1.566361322s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
10312025-04-30T06:38:09.535ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = region_replacement elapsed = 1.591562424s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10322025-04-30T06:38:09.655ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10332025-04-30T06:38:09.655ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10342025-04-30T06:38:09.655ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10352025-04-30T06:38:09.655ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10362025-04-30T06:38:09.655ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10372025-04-30T06:38:09.656ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10382025-04-30T06:38:09.656ZINFOtest_quotas (dropshot_internal): request completed latency_us = 1549784 local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:60161 req_id = f35e448e-bb40-444c-ac9b-8067f9ea9af5 response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
10392025-04-30T06:38:09.657ZDEBGtest_quotas (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:64801/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "f35e448e-bb40-444c-ac9b-8067f9ea9af5", "date": "Wed, 30 Apr 2025 06:38:09 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10402025-04-30T06:38:09.657ZDEBGtest_quotas (init_with_steps): Running step start_crucible_pantry
10412025-04-30T06:38:09.659ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10422025-04-30T06:38:09.659ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10432025-04-30T06:38:09.660ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10442025-04-30T06:38:09.660ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10452025-04-30T06:38:09.660ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10462025-04-30T06:38:09.660ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10472025-04-30T06:38:09.660ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10482025-04-30T06:38:09.660ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10492025-04-30T06:38:09.660ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10502025-04-30T06:38:09.660ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10512025-04-30T06:38:09.660ZDEBGtest_quotas (dropshot): registered endpoint kind = pantry local_addr = [::1]:40067 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10522025-04-30T06:38:09.660ZINFOtest_quotas (dropshot): listening kind = pantry local_addr = [::1]:40067 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10532025-04-30T06:38:09.660ZDEBGtest_quotas (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:40067 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10542025-04-30T06:38:09.660ZINFOtest_quotas (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:40067 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10552025-04-30T06:38:09.661ZDEBGtest_quotas (init_with_steps): Running step populate_internal_dns
10562025-04-30T06:38:09.661ZDEBGtest_quotas: Populating Internal DNS
10572025-04-30T06:38:09.705ZINFOtest_quotas: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:38:09.705550480Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 56554, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "f0fa6243-1db8-401f-9a4d-243fcac463b4.host": [ Aaaa( ::1, ), ], "503d917a-5d80-4b21-9607-0ae96af250a8.host": [ Aaaa( ::1, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 39362, target: "e0da77c8-3edf-44ee-81a0-e09b1bc94252.host.control-plane.oxide.internal", }, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 64801, target: "4678dd50-d1e0-4228-8533-c4090342ff58.host.control-plane.oxide.internal", }, ), ], "e0da77c8-3edf-44ee-81a0-e09b1bc94252.host": [ Aaaa( ::1, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "e0da77c8-3edf-44ee-81a0-e09b1bc94252.host.control-plane.oxide.internal", }, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "e0da77c8-3edf-44ee-81a0-e09b1bc94252.host.control-plane.oxide.internal", }, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 40067, target: "f0fa6243-1db8-401f-9a4d-243fcac463b4.host.control-plane.oxide.internal", }, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "4678dd50-d1e0-4228-8533-c4090342ff58.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host": [ Aaaa( ::1, ), ], "5df32d55-14c3-4a54-8cf5-6366e3f60c12.host": [ Aaaa( ::1, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 44150, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "e0da77c8-3edf-44ee-81a0-e09b1bc94252.host.control-plane.oxide.internal", }, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 61168, target: "5df32d55-14c3-4a54-8cf5-6366e3f60c12.host.control-plane.oxide.internal", }, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 57119, target: "4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 36358, target: "503d917a-5d80-4b21-9607-0ae96af250a8.host.control-plane.oxide.internal", }, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 55999, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 39922, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], }, }, ], }
10582025-04-30T06:38:09.706ZDEBGtest_quotas: client request body = Some(Body) method = PUT uri = http://[::1]:57119/config
10592025-04-30T06:38:09.706ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
10602025-04-30T06:38:09.707ZDEBGtest_quotas (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10612025-04-30T06:38:09.708ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10622025-04-30T06:38:09.708ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 1.751621281s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10632025-04-30T06:38:09.708ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10642025-04-30T06:38:09.709ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10652025-04-30T06:38:09.709ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10662025-04-30T06:38:09.709ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10672025-04-30T06:38:09.710ZWARNtest_quotas (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
10682025-04-30T06:38:09.710ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 1.789709379s iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
10692025-04-30T06:38:09.710ZINFOtest_quotas (http): accepted connection kind = dns local_addr = [::1]:57119 remote_addr = [::1]:43047
10702025-04-30T06:38:09.712ZTRACtest_quotas (http): incoming request kind = dns local_addr = [::1]:57119 method = PUT remote_addr = [::1]:43047 req_id = 887aff08-8e5d-42f1-80c1-4ff738643b24 uri = /config
10712025-04-30T06:38:09.713ZINFOtest_quotas (store): attempting generation update new_generation = 1 req_id = 887aff08-8e5d-42f1-80c1-4ff738643b24
10722025-04-30T06:38:09.713ZINFOtest_quotas (store): pruning trees for generations newer than 0
10732025-04-30T06:38:09.713ZDEBGtest_quotas (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
10742025-04-30T06:38:09.714ZDEBGtest_quotas (store): updating current config
10752025-04-30T06:38:09.714ZDEBGtest_quotas (store): flushing default tree
10762025-04-30T06:38:09.714ZINFOtest_quotas (store): pruning trees for generations older than 1 keep = 3
10772025-04-30T06:38:09.714ZINFOtest_quotas (store): updated generation new_generation = 1 req_id = 887aff08-8e5d-42f1-80c1-4ff738643b24
10782025-04-30T06:38:09.714ZINFOtest_quotas (http): request completed kind = dns latency_us = 2572 local_addr = [::1]:57119 method = PUT remote_addr = [::1]:43047 req_id = 887aff08-8e5d-42f1-80c1-4ff738643b24 response_code = 204 uri = /config
10792025-04-30T06:38:09.715ZDEBGtest_quotas: client response result = Ok(Response { url: "http://[::1]:57119/config", status: 204, headers: {"x-request-id": "887aff08-8e5d-42f1-80c1-4ff738643b24", "date": "Wed, 30 Apr 2025 06:38:09 GMT"} })
10802025-04-30T06:38:09.716ZDEBGtest_quotas (init_with_steps): Running step configure_sled_agents
10812025-04-30T06:38:09.768ZDEBGtest_quotas: client request body = Some(Body) method = PUT uri = http://[::1]:55655/omicron-config
10822025-04-30T06:38:09.768ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:43293 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10832025-04-30T06:38:09.769ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = PUT remote_addr = [::1]:43293 req_id = 029fb42f-4c32-442f-b7c3-f08c6ef08c46 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10842025-04-30T06:38:09.772ZINFOtest_quotas (dropshot): request completed latency_us = 3540 local_addr = [::1]:55655 method = PUT remote_addr = [::1]:43293 req_id = 029fb42f-4c32-442f-b7c3-f08c6ef08c46 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10852025-04-30T06:38:09.774ZDEBGtest_quotas: client response result = Ok(Response { url: "http://[::1]:55655/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "029fb42f-4c32-442f-b7c3-f08c6ef08c46", "content-length": "26", "date": "Wed, 30 Apr 2025 06:38:09 GMT"} })
10862025-04-30T06:38:09.775ZDEBGtest_quotas: client request body = Some(Body) method = PUT uri = http://[::1]:55655/network-bootstore-config
10872025-04-30T06:38:09.776ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = PUT remote_addr = [::1]:43293 req_id = e6890bce-dfa5-4c90-99a5-558f7b69ee5e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10882025-04-30T06:38:09.777ZINFOtest_quotas (dropshot): request completed latency_us = 1207 local_addr = [::1]:55655 method = PUT remote_addr = [::1]:43293 req_id = e6890bce-dfa5-4c90-99a5-558f7b69ee5e response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10892025-04-30T06:38:09.777ZDEBGtest_quotas: client response result = Ok(Response { url: "http://[::1]:55655/network-bootstore-config", status: 204, headers: {"x-request-id": "e6890bce-dfa5-4c90-99a5-558f7b69ee5e", "date": "Wed, 30 Apr 2025 06:38:09 GMT"} })
10902025-04-30T06:38:09.778ZDEBGtest_quotas (init_with_steps): Running step start_nexus_external
10912025-04-30T06:38:09.778ZDEBGtest_quotas: Starting Nexus (external API)
10922025-04-30T06:38:12.886ZTRACtest_quotas (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
10932025-04-30T06:38:12.892ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10942025-04-30T06:38:12.892ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
10952025-04-30T06:38:12.892ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
10962025-04-30T06:38:12.893ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
10972025-04-30T06:38:12.893ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10982025-04-30T06:38:12.898ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
10992025-04-30T06:38:12.898ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
11002025-04-30T06:38:12.898ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
11012025-04-30T06:38:12.898ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
11022025-04-30T06:38:12.898ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11032025-04-30T06:38:12.908ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
11042025-04-30T06:38:12.908ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
11052025-04-30T06:38:12.908ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11062025-04-30T06:38:12.909ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11072025-04-30T06:38:12.909ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
11082025-04-30T06:38:12.909ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11092025-04-30T06:38:12.909ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
11102025-04-30T06:38:12.909ZDEBGtest_quotas (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11112025-04-30T06:38:12.910ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11122025-04-30T06:38:12.912ZWARNtest_quotas (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
11132025-04-30T06:38:12.912ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 4.987657811s iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
11142025-04-30T06:38:12.983ZINFOtest_quotas (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
11152025-04-30T06:38:12.983ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:55655/vpc-routes
11162025-04-30T06:38:12.984ZTRACtest_quotas (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
11172025-04-30T06:38:12.984ZINFOtest_quotas (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11182025-04-30T06:38:12.984ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 5.034070937s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11192025-04-30T06:38:12.984ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11202025-04-30T06:38:12.984ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11212025-04-30T06:38:12.984ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11222025-04-30T06:38:12.984ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11232025-04-30T06:38:12.985ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11242025-04-30T06:38:13.036ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11252025-04-30T06:38:13.036ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11262025-04-30T06:38:13.036ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11272025-04-30T06:38:13.036ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11282025-04-30T06:38:13.037ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11292025-04-30T06:38:13.038ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 5.080885933s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11302025-04-30T06:38:13.040ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11312025-04-30T06:38:13.049ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11322025-04-30T06:38:13.054ZTRACtest_quotas (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
11332025-04-30T06:38:13.054ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
11342025-04-30T06:38:13.054ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
11352025-04-30T06:38:13.054ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11362025-04-30T06:38:13.054ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11372025-04-30T06:38:13.055ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11382025-04-30T06:38:13.055ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:56734 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11392025-04-30T06:38:13.058ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:56734 req_id = ae06d9ea-29b7-4201-bd03-95817483854d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11402025-04-30T06:38:13.059ZINFOtest_quotas (dropshot): request completed latency_us = 613 local_addr = [::1]:55655 method = GET remote_addr = [::1]:56734 req_id = ae06d9ea-29b7-4201-bd03-95817483854d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11412025-04-30T06:38:13.071ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:55655/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae06d9ea-29b7-4201-bd03-95817483854d", "content-length": "2", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
11422025-04-30T06:38:13.071ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
11432025-04-30T06:38:13.073ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11442025-04-30T06:38:13.074ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11452025-04-30T06:38:13.075ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11462025-04-30T06:38:13.079ZDEBGtest_quotas (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
11472025-04-30T06:38:13.081ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11482025-04-30T06:38:13.081ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
11492025-04-30T06:38:13.081ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11502025-04-30T06:38:13.083ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11512025-04-30T06:38:13.083ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11522025-04-30T06:38:13.091ZDEBGtest_quotas (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11532025-04-30T06:38:13.109ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11542025-04-30T06:38:13.109ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
11552025-04-30T06:38:13.109ZDEBGtest_quotas (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11562025-04-30T06:38:13.111ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11572025-04-30T06:38:13.111ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11582025-04-30T06:38:13.116ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11592025-04-30T06:38:13.116ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11602025-04-30T06:38:13.116ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11612025-04-30T06:38:13.116ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11622025-04-30T06:38:13.116ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11632025-04-30T06:38:13.117ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11642025-04-30T06:38:13.118ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 94 3a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .:..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53723 req_id = 018903d0-f99e-4de6-92d2-b1aee05a8958
11652025-04-30T06:38:13.118ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37946,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53723 req_id = 018903d0-f99e-4de6-92d2-b1aee05a8958
11662025-04-30T06:38:13.119ZDEBGtest_quotas (store): query key key = _mgs._tcp
11672025-04-30T06:38:13.119ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11682025-04-30T06:38:13.119ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53723 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 018903d0-f99e-4de6-92d2-b1aee05a8958
11692025-04-30T06:38:13.122ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11702025-04-30T06:38:13.142ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11712025-04-30T06:38:13.142ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11722025-04-30T06:38:13.142ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11732025-04-30T06:38:13.143ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11742025-04-30T06:38:13.143ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11752025-04-30T06:38:13.143ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11762025-04-30T06:38:13.143ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11772025-04-30T06:38:13.144ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11782025-04-30T06:38:13.150ZWARNtest_quotas (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
11792025-04-30T06:38:13.150ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 5.222975437s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
11802025-04-30T06:38:13.152ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3957, tv_nsec: 950964169 } })
11812025-04-30T06:38:13.152ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
11822025-04-30T06:38:13.172ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11832025-04-30T06:38:13.172ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
11842025-04-30T06:38:13.178ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11852025-04-30T06:38:13.180ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5c a0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e \\...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63209 req_id = fe0c4152-f2b1-4d85-87d6-a71ce1a06919
11862025-04-30T06:38:13.180ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23712,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63209 req_id = fe0c4152-f2b1-4d85-87d6-a71ce1a06919
11872025-04-30T06:38:13.180ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11882025-04-30T06:38:13.180ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63209 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fe0c4152-f2b1-4d85-87d6-a71ce1a06919
11892025-04-30T06:38:13.242ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
11902025-04-30T06:38:13.247ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11912025-04-30T06:38:13.254ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11922025-04-30T06:38:13.254ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11932025-04-30T06:38:13.254ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11942025-04-30T06:38:13.256ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11952025-04-30T06:38:13.257ZINFOtest_quotas (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
11962025-04-30T06:38:13.257ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:55655/eip-gateways
11972025-04-30T06:38:13.258ZINFOtest_quotas (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
11982025-04-30T06:38:13.258ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 5.324248697s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11992025-04-30T06:38:13.258ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
12002025-04-30T06:38:13.258ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12012025-04-30T06:38:13.258ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12022025-04-30T06:38:13.258ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12032025-04-30T06:38:13.259ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12042025-04-30T06:38:13.260ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12052025-04-30T06:38:13.260ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12062025-04-30T06:38:13.260ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12072025-04-30T06:38:13.261ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12082025-04-30T06:38:13.262ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = PUT remote_addr = [::1]:56734 req_id = 1158571d-cf2a-47f6-aa99-67878778c2a8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12092025-04-30T06:38:13.269ZINFOtest_quotas (dropshot): request completed latency_us = 6593 local_addr = [::1]:55655 method = PUT remote_addr = [::1]:56734 req_id = 1158571d-cf2a-47f6-aa99-67878778c2a8 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12102025-04-30T06:38:13.271ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 69 33 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c i3..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54110 req_id = 04fa0535-ceab-46d7-9184-8e3e45ed25f0
12112025-04-30T06:38:13.271ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26931,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54110 req_id = 04fa0535-ceab-46d7-9184-8e3e45ed25f0
12122025-04-30T06:38:13.271ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
12132025-04-30T06:38:13.271ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26931, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54110 req_id = 04fa0535-ceab-46d7-9184-8e3e45ed25f0
12142025-04-30T06:38:13.272ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:55655/eip-gateways", status: 204, headers: {"x-request-id": "1158571d-cf2a-47f6-aa99-67878778c2a8", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
12152025-04-30T06:38:13.273ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 5.320072637s iteration = 1 status = {}
12162025-04-30T06:38:13.277ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
12172025-04-30T06:38:13.277ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
12182025-04-30T06:38:13.278ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
12192025-04-30T06:38:13.283ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12202025-04-30T06:38:13.289ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12212025-04-30T06:38:13.289ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12222025-04-30T06:38:13.289ZDEBGtest_quotas (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12232025-04-30T06:38:13.289ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12242025-04-30T06:38:13.290ZINFOtest_quotas (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
12252025-04-30T06:38:13.291ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12262025-04-30T06:38:13.291ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
12272025-04-30T06:38:13.291ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12282025-04-30T06:38:13.297ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12292025-04-30T06:38:13.306ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12302025-04-30T06:38:13.307ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12312025-04-30T06:38:13.307ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12322025-04-30T06:38:13.307ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12332025-04-30T06:38:13.308ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:55655/artifacts-config
12342025-04-30T06:38:13.310ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:54434 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12352025-04-30T06:38:13.317ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12362025-04-30T06:38:13.317ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12372025-04-30T06:38:13.318ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12382025-04-30T06:38:13.320ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
12392025-04-30T06:38:13.320ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12402025-04-30T06:38:13.320ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12412025-04-30T06:38:13.320ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
12422025-04-30T06:38:13.321ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12432025-04-30T06:38:13.324ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = PUT remote_addr = [::1]:54434 req_id = 0e53fb3a-80d9-4090-90c3-e8fa1f774010 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12442025-04-30T06:38:13.336ZINFOtest_quotas (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp3gqMPc/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12452025-04-30T06:38:13.336ZDEBGtest_quotas (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp3gqMPc/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12462025-04-30T06:38:13.336ZINFOtest_quotas (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpsOhJx1/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12472025-04-30T06:38:13.336ZDEBGtest_quotas (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpsOhJx1/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12482025-04-30T06:38:13.336ZINFOtest_quotas (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmp3gqMPc/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12492025-04-30T06:38:13.337ZINFOtest_quotas (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpsOhJx1/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12502025-04-30T06:38:13.338ZINFOtest_quotas (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12512025-04-30T06:38:13.339ZINFOtest_quotas (dropshot): request completed latency_us = 14936 local_addr = [::1]:55655 method = PUT remote_addr = [::1]:54434 req_id = 0e53fb3a-80d9-4090-90c3-e8fa1f774010 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12522025-04-30T06:38:13.344ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:55655/artifacts-config", status: 204, headers: {"x-request-id": "0e53fb3a-80d9-4090-90c3-e8fa1f774010", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
12532025-04-30T06:38:13.344ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:55655/artifacts
12542025-04-30T06:38:13.345ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12552025-04-30T06:38:13.345ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12562025-04-30T06:38:13.346ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12572025-04-30T06:38:13.353ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:54434 req_id = cf467574-9f10-4a52-9f10-7f3312e1dc42 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12582025-04-30T06:38:13.353ZINFOtest_quotas (dropshot): request completed latency_us = 470 local_addr = [::1]:55655 method = GET remote_addr = [::1]:54434 req_id = cf467574-9f10-4a52-9f10-7f3312e1dc42 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12592025-04-30T06:38:13.354ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12602025-04-30T06:38:13.411ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
12612025-04-30T06:38:13.411ZDEBGtest_quotas (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
12622025-04-30T06:38:13.411ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12632025-04-30T06:38:13.411ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
12642025-04-30T06:38:13.412ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12652025-04-30T06:38:13.419ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:55655/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf467574-9f10-4a52-9f10-7f3312e1dc42", "content-length": "26", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
12662025-04-30T06:38:13.419ZINFOtest_quotas (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:55655
12672025-04-30T06:38:13.420ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12682025-04-30T06:38:13.420ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12692025-04-30T06:38:13.421ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12702025-04-30T06:38:13.426ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 5.466659462s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:38:13.344348286Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:38:13.419961852Z"}]}
12712025-04-30T06:38:13.482ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:55655/v2p
12722025-04-30T06:38:13.492ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:40388 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12732025-04-30T06:38:13.492ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7df80725-e42e-451f-8edf-1bfbbbb062bd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
12742025-04-30T06:38:13.493ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
12752025-04-30T06:38:13.494ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12762025-04-30T06:38:13.494ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12772025-04-30T06:38:13.494ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12782025-04-30T06:38:13.494ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12792025-04-30T06:38:13.495ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12802025-04-30T06:38:13.497ZINFOtest_quotas (InternalApi): Using static configuration for external switchports
12812025-04-30T06:38:13.497ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12822025-04-30T06:38:13.497ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
12832025-04-30T06:38:13.498ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12842025-04-30T06:38:13.499ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:40388 req_id = 73527ef4-316f-4f40-ae33-81b6e22b9109 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12852025-04-30T06:38:13.501ZINFOtest_quotas (dropshot): request completed latency_us = 1992 local_addr = [::1]:55655 method = GET remote_addr = [::1]:40388 req_id = 73527ef4-316f-4f40-ae33-81b6e22b9109 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12862025-04-30T06:38:13.503ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:55655/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "73527ef4-316f-4f40-ae33-81b6e22b9109", "content-length": "2", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
12872025-04-30T06:38:13.503ZINFOtest_quotas (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
12882025-04-30T06:38:13.503ZINFOtest_quotas (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
12892025-04-30T06:38:13.504ZINFOtest_quotas (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
12902025-04-30T06:38:13.505ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 5.567046316s iteration = 1 status = {}
12912025-04-30T06:38:13.507ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12922025-04-30T06:38:13.507ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12932025-04-30T06:38:13.507ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12942025-04-30T06:38:13.508ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12952025-04-30T06:38:13.526ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12962025-04-30T06:38:13.526ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12972025-04-30T06:38:13.526ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12982025-04-30T06:38:13.527ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12992025-04-30T06:38:13.533ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13002025-04-30T06:38:13.553ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13012025-04-30T06:38:13.553ZTRACtest_quotas (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
13022025-04-30T06:38:13.554ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13032025-04-30T06:38:13.554ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13042025-04-30T06:38:13.555ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13052025-04-30T06:38:13.566ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13062025-04-30T06:38:13.566ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13072025-04-30T06:38:13.567ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13082025-04-30T06:38:13.569ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e2a01e7-c5c1-4186-ad14-fefcb63bbd87", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
13092025-04-30T06:38:13.570ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
13102025-04-30T06:38:13.580ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13112025-04-30T06:38:13.580ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13122025-04-30T06:38:13.581ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13132025-04-30T06:38:13.616ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13142025-04-30T06:38:13.616ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13152025-04-30T06:38:13.617ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13162025-04-30T06:38:13.620ZINFOtest_quotas (InternalApi): Recording Rack Network Configuration
13172025-04-30T06:38:13.620ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13182025-04-30T06:38:13.620ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
13192025-04-30T06:38:13.621ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13202025-04-30T06:38:13.630ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13212025-04-30T06:38:13.630ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13222025-04-30T06:38:13.630ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13232025-04-30T06:38:13.630ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13242025-04-30T06:38:13.631ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13252025-04-30T06:38:13.636ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13262025-04-30T06:38:13.636ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13272025-04-30T06:38:13.636ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-04-30T06:38:13.637ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-04-30T06:38:13.642ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13302025-04-30T06:38:13.642ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13312025-04-30T06:38:13.642ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13322025-04-30T06:38:13.643ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13332025-04-30T06:38:13.650ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13342025-04-30T06:38:13.658ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
13352025-04-30T06:38:13.658ZTRACtest_quotas (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13362025-04-30T06:38:13.658ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13372025-04-30T06:38:13.658ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13382025-04-30T06:38:13.658ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13392025-04-30T06:38:13.658ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13402025-04-30T06:38:13.658ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13412025-04-30T06:38:13.661ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13422025-04-30T06:38:13.661ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13432025-04-30T06:38:13.661ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13442025-04-30T06:38:13.662ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13452025-04-30T06:38:13.664ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13462025-04-30T06:38:13.664ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13472025-04-30T06:38:13.664ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13482025-04-30T06:38:13.665ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13492025-04-30T06:38:13.670ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13502025-04-30T06:38:13.676ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
13512025-04-30T06:38:13.676ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13522025-04-30T06:38:13.676ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13532025-04-30T06:38:13.677ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13542025-04-30T06:38:13.688ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13552025-04-30T06:38:13.688ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13562025-04-30T06:38:13.688ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13572025-04-30T06:38:13.689ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83275897-951e-41c4-8ca7-dfadaf891d8e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
13582025-04-30T06:38:13.689ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
13592025-04-30T06:38:13.691ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13602025-04-30T06:38:13.691ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13612025-04-30T06:38:13.692ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13622025-04-30T06:38:13.697ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13632025-04-30T06:38:13.698ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13642025-04-30T06:38:13.698ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13652025-04-30T06:38:13.703ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13662025-04-30T06:38:13.703ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13672025-04-30T06:38:13.704ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13682025-04-30T06:38:13.708ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13692025-04-30T06:38:13.708ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13702025-04-30T06:38:13.708ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13712025-04-30T06:38:13.708ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13722025-04-30T06:38:13.708ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13732025-04-30T06:38:13.720ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13742025-04-30T06:38:13.720ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13752025-04-30T06:38:13.721ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13762025-04-30T06:38:13.721ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13772025-04-30T06:38:13.723ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6be09d74-d6c4-4050-b259-dca352026878", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
13782025-04-30T06:38:13.724ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
13792025-04-30T06:38:13.729ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13802025-04-30T06:38:13.729ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13812025-04-30T06:38:13.729ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13822025-04-30T06:38:13.729ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13832025-04-30T06:38:13.734ZTRACtest_quotas (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
13842025-04-30T06:38:13.734ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13852025-04-30T06:38:13.734ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13862025-04-30T06:38:13.734ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
13872025-04-30T06:38:13.735ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13882025-04-30T06:38:13.736ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13892025-04-30T06:38:13.742ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13902025-04-30T06:38:13.742ZTRACtest_quotas (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13912025-04-30T06:38:13.742ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13922025-04-30T06:38:13.742ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13932025-04-30T06:38:13.742ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13942025-04-30T06:38:13.742ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13952025-04-30T06:38:13.743ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13962025-04-30T06:38:13.745ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13972025-04-30T06:38:13.748ZDEBGtest_quotas (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
13982025-04-30T06:38:13.748ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13992025-04-30T06:38:13.748ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
14002025-04-30T06:38:13.749ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14012025-04-30T06:38:13.749ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14022025-04-30T06:38:13.749ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14032025-04-30T06:38:13.749ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14042025-04-30T06:38:13.750ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14052025-04-30T06:38:13.755ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 998a39c9-49c7-4e2f-ad21-0fb0ad20281e, lookup_type: ByName("oxide-service-pool") }
14062025-04-30T06:38:13.755ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14072025-04-30T06:38:13.755ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14082025-04-30T06:38:13.755ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
14092025-04-30T06:38:13.757ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14102025-04-30T06:38:13.757ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14112025-04-30T06:38:13.757ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14122025-04-30T06:38:13.757ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14132025-04-30T06:38:13.758ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14142025-04-30T06:38:13.760ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14152025-04-30T06:38:13.765ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 998a39c9-49c7-4e2f-ad21-0fb0ad20281e, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14162025-04-30T06:38:13.765ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14172025-04-30T06:38:13.765ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
14182025-04-30T06:38:13.766ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14192025-04-30T06:38:13.766ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14202025-04-30T06:38:13.775ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
14212025-04-30T06:38:13.775ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14222025-04-30T06:38:13.775ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14232025-04-30T06:38:13.776ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14242025-04-30T06:38:13.786ZTRACtest_quotas (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 998a39c9-49c7-4e2f-ad21-0fb0ad20281e, lookup_type: ByName("oxide-service-pool") }
14252025-04-30T06:38:13.786ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14262025-04-30T06:38:13.786ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14272025-04-30T06:38:13.786ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
14282025-04-30T06:38:13.787ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14292025-04-30T06:38:13.796ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14302025-04-30T06:38:13.803ZDEBGtest_quotas (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 998a39c9-49c7-4e2f-ad21-0fb0ad20281e, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14312025-04-30T06:38:13.804ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92099cb3-5846-4e91-8766-987fd95c1e65", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14322025-04-30T06:38:13.804ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
14332025-04-30T06:38:13.805ZDEBGtest_quotas (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
14342025-04-30T06:38:13.805ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
14352025-04-30T06:38:13.805ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14362025-04-30T06:38:13.805ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14372025-04-30T06:38:13.805ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14382025-04-30T06:38:13.806ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14392025-04-30T06:38:13.809ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14402025-04-30T06:38:13.818ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
14412025-04-30T06:38:13.818ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14422025-04-30T06:38:13.818ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14432025-04-30T06:38:13.819ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14442025-04-30T06:38:13.820ZTRACtest_quotas (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 998a39c9-49c7-4e2f-ad21-0fb0ad20281e, lookup_type: ByName("oxide-service-pool") }
14452025-04-30T06:38:13.821ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14462025-04-30T06:38:13.821ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14472025-04-30T06:38:13.821ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
14482025-04-30T06:38:13.821ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14492025-04-30T06:38:13.824ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e86e5ad-7dce-4ba9-8c48-78dad5c46c16", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14502025-04-30T06:38:13.825ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
14512025-04-30T06:38:13.825ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14522025-04-30T06:38:13.834ZDEBGtest_quotas (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 998a39c9-49c7-4e2f-ad21-0fb0ad20281e, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14532025-04-30T06:38:13.834ZTRACtest_quotas (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
14542025-04-30T06:38:13.834ZDEBGtest_quotas (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
14552025-04-30T06:38:13.834ZDEBGtest_quotas (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
14562025-04-30T06:38:13.835ZDEBGtest_quotas (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
14572025-04-30T06:38:13.839ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9f1dbf5-040e-49db-b605-bc558af88f73", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14582025-04-30T06:38:13.839ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
14592025-04-30T06:38:13.840ZTRACtest_quotas (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14602025-04-30T06:38:13.840ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14612025-04-30T06:38:13.840ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14622025-04-30T06:38:13.840ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
14632025-04-30T06:38:13.841ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14642025-04-30T06:38:13.843ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14652025-04-30T06:38:13.848ZDEBGtest_quotas (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
14662025-04-30T06:38:13.848ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa14e56d-3a53-4b1a-8c46-2915a578ff4c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14672025-04-30T06:38:13.848ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
14682025-04-30T06:38:13.850ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d8d9702-d5d3-4214-a3da-b06c77aa3ec0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14692025-04-30T06:38:13.850ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
14702025-04-30T06:38:13.852ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb1584c7-7a75-4949-83c8-b259ce563dce", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14712025-04-30T06:38:13.853ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
14722025-04-30T06:38:13.861ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1e90a1f-de41-44eb-ae34-05dfa58e919c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14732025-04-30T06:38:13.867ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
14742025-04-30T06:38:13.890ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca467bc2-0613-42c5-aa96-ffc7461975a1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14752025-04-30T06:38:13.890ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
14762025-04-30T06:38:13.892ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "426cac6e-1f35-428b-a552-422a87301610", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14772025-04-30T06:38:13.892ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
14782025-04-30T06:38:13.896ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2def2452-447b-42f7-9654-62a4a289eba7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14792025-04-30T06:38:13.896ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
14802025-04-30T06:38:13.900ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d281bdd-be78-4252-8bad-52e26ea88a4b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14812025-04-30T06:38:13.900ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
14822025-04-30T06:38:13.903ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdeb3681-7829-4fee-b078-b65a2dcf820b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14832025-04-30T06:38:13.903ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
14842025-04-30T06:38:13.909ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "798cdb74-9bc5-4195-9cee-2250004c26f9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14852025-04-30T06:38:13.910ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
14862025-04-30T06:38:13.915ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3aaeb5e-5e4c-454c-9498-3f6b585701aa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14872025-04-30T06:38:13.915ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
14882025-04-30T06:38:13.921ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e559d927-d407-46d4-8946-92f2b6a5073d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14892025-04-30T06:38:13.921ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
14902025-04-30T06:38:13.936ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7809fc34-a75b-4527-97b9-d8bb3a4b38cb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14912025-04-30T06:38:13.936ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
14922025-04-30T06:38:13.947ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb795bec-c391-4b82-9ed9-2c472c50a9ba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14932025-04-30T06:38:13.947ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
14942025-04-30T06:38:13.992ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48da2510-ff36-4bb9-9b1b-41528cc9a3c8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14952025-04-30T06:38:13.992ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
14962025-04-30T06:38:14.003ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c42efb40-be0e-4007-bbfd-61ea77aa4f02", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14972025-04-30T06:38:14.003ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
14982025-04-30T06:38:14.004ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc745571-4f22-42b5-a62e-6341704115e6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
14992025-04-30T06:38:14.005ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
15002025-04-30T06:38:14.007ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d1e726a-749b-4017-aeda-0a91dd662acc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15012025-04-30T06:38:14.007ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
15022025-04-30T06:38:14.009ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b3f79e6-e279-44cc-8dfd-f661e7035dba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15032025-04-30T06:38:14.010ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
15042025-04-30T06:38:14.012ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d97bde1-de12-4934-a12c-ac16f2e0de4d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15052025-04-30T06:38:14.012ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
15062025-04-30T06:38:14.017ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23da8732-ec3b-4ad6-8311-246b4192743a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15072025-04-30T06:38:14.017ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
15082025-04-30T06:38:14.020ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed742d37-0c1c-4778-99ef-a97429df0543", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15092025-04-30T06:38:14.020ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
15102025-04-30T06:38:14.024ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ec0ace2-1e1d-4373-b50e-fa4579f6604d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15112025-04-30T06:38:14.024ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
15122025-04-30T06:38:14.025ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcbbde17-f14f-48aa-b86d-a4cb79610bc7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15132025-04-30T06:38:14.025ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
15142025-04-30T06:38:14.037ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae20f799-978b-4022-8972-adb2542d8d14", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15152025-04-30T06:38:14.037ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
15162025-04-30T06:38:14.039ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "858783f3-6978-4dab-abaf-c379249cbdb5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15172025-04-30T06:38:14.039ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
15182025-04-30T06:38:14.041ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c03299d6-4340-476d-9f7f-fb3a1d9bb718", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15192025-04-30T06:38:14.042ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
15202025-04-30T06:38:14.044ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1be49661-616b-4b3a-9a6e-1707749f5664", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:13 GMT"} })
15212025-04-30T06:38:14.045ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
15222025-04-30T06:38:14.067ZDEBGtest_quotas (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
15232025-04-30T06:38:14.067ZDEBGtest_quotas (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
15242025-04-30T06:38:14.067ZINFOtest_quotas (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
15252025-04-30T06:38:14.067ZDEBGtest_quotas (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 6.115155006s
15262025-04-30T06:38:14.068ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 6.115264515s iteration = 1 status = {}
15272025-04-30T06:38:14.069ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84826bd2-5d4a-4938-bbfd-736d6725acd3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15282025-04-30T06:38:14.073ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
15292025-04-30T06:38:14.087ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a894aed-6fae-464f-adb3-a1f618f0f5be", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15302025-04-30T06:38:14.087ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
15312025-04-30T06:38:14.092ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e46a076-7f67-474e-b5b5-9f121cc9e0aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15322025-04-30T06:38:14.092ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
15332025-04-30T06:38:14.095ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "527743b6-db02-4dae-baa7-bfd5e30abf52", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15342025-04-30T06:38:14.095ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
15352025-04-30T06:38:14.096ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c849545d-d13c-403d-9d90-0779bb6558b8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15362025-04-30T06:38:14.096ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
15372025-04-30T06:38:14.098ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5dfcc5-c857-47a5-b328-bb32309d799f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15382025-04-30T06:38:14.099ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
15392025-04-30T06:38:14.099ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "320c5c12-f5a8-472f-99d4-fa0fb2d797df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15402025-04-30T06:38:14.100ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
15412025-04-30T06:38:14.101ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12256fd9-612d-4795-ab29-2586a82c6797", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15422025-04-30T06:38:14.101ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
15432025-04-30T06:38:14.102ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17d889e4-523b-4398-a4a2-43fb80172bbf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15442025-04-30T06:38:14.102ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
15452025-04-30T06:38:14.103ZINFOtest_quotas (InternalApi): inserted blueprint blueprint_id = 38254a39-b584-4d35-98a8-78685f2aa081
15462025-04-30T06:38:14.103ZTRACtest_quotas (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
15472025-04-30T06:38:14.103ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15482025-04-30T06:38:14.103ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15492025-04-30T06:38:14.103ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
15502025-04-30T06:38:14.104ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15512025-04-30T06:38:14.108ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea83cde8-9d92-4e47-b897-3532cb4f1482", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15522025-04-30T06:38:14.115ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15532025-04-30T06:38:14.115ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15542025-04-30T06:38:14.116ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15552025-04-30T06:38:14.116ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15562025-04-30T06:38:14.116ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15572025-04-30T06:38:14.128ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15582025-04-30T06:38:14.133ZDEBGtest_quotas (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
15592025-04-30T06:38:14.165ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15602025-04-30T06:38:14.171ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15612025-04-30T06:38:14.171ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15622025-04-30T06:38:14.171ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15632025-04-30T06:38:14.171ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15642025-04-30T06:38:14.171ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15652025-04-30T06:38:14.172ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15662025-04-30T06:38:14.184ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15672025-04-30T06:38:14.193ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15682025-04-30T06:38:14.193ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15692025-04-30T06:38:14.193ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15702025-04-30T06:38:14.194ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15712025-04-30T06:38:14.356ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
15722025-04-30T06:38:14.356ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
15732025-04-30T06:38:14.358ZINFOtest_quotas (InternalApi): Inserted networking records for nexus service
15742025-04-30T06:38:14.358ZINFOtest_quotas (InternalApi): No networking records needed for internal_dns service
15752025-04-30T06:38:14.359ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:42386 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15762025-04-30T06:38:14.360ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:42386 req_id = 780aa1c9-cbe5-4256-96ee-5081ac54fbef sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15772025-04-30T06:38:14.373ZINFOtest_quotas (dropshot): request completed latency_us = 12589 local_addr = [::1]:55655 method = GET remote_addr = [::1]:42386 req_id = 780aa1c9-cbe5-4256-96ee-5081ac54fbef response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15782025-04-30T06:38:14.373ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "780aa1c9-cbe5-4256-96ee-5081ac54fbef", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:38:14 GMT"} })
15792025-04-30T06:38:14.376ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15802025-04-30T06:38:14.376ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15812025-04-30T06:38:14.376ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
15822025-04-30T06:38:14.378ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
15832025-04-30T06:38:14.378ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15842025-04-30T06:38:14.378ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15852025-04-30T06:38:14.378ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15862025-04-30T06:38:14.379ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15872025-04-30T06:38:14.390ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15882025-04-30T06:38:14.399ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
15892025-04-30T06:38:14.399ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15902025-04-30T06:38:14.399ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15912025-04-30T06:38:14.400ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15922025-04-30T06:38:14.401ZINFOtest_quotas (InternalApi): Inserted networking records for external_dns service
15932025-04-30T06:38:14.401ZINFOtest_quotas (InternalApi): No networking records needed for cockroach_db service
15942025-04-30T06:38:14.401ZINFOtest_quotas (InternalApi): No networking records needed for clickhouse service
15952025-04-30T06:38:14.401ZINFOtest_quotas (InternalApi): No networking records needed for crucible_pantry service
15962025-04-30T06:38:14.401ZINFOtest_quotas (InternalApi): Inserted service networking records
15972025-04-30T06:38:14.401ZINFOtest_quotas (InternalApi): Inserted physical disks
15982025-04-30T06:38:14.401ZINFOtest_quotas (InternalApi): Inserted zpools
15992025-04-30T06:38:14.401ZINFOtest_quotas (InternalApi): Inserted datasets
16002025-04-30T06:38:14.462ZINFOtest_quotas (InternalApi): Populated DNS tables for internal DNS
16012025-04-30T06:38:14.484ZINFOtest_quotas (InternalApi): Populated DNS tables for external DNS
16022025-04-30T06:38:14.484ZTRACtest_quotas (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16032025-04-30T06:38:14.484ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16042025-04-30T06:38:14.484ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16052025-04-30T06:38:14.484ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16062025-04-30T06:38:14.486ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16072025-04-30T06:38:14.488ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16082025-04-30T06:38:14.499ZDEBGtest_quotas (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16092025-04-30T06:38:14.500ZTRACtest_quotas (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16102025-04-30T06:38:14.500ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16112025-04-30T06:38:14.500ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16122025-04-30T06:38:14.500ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16132025-04-30T06:38:14.500ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16142025-04-30T06:38:14.508ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16152025-04-30T06:38:14.512ZDEBGtest_quotas (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16162025-04-30T06:38:14.545ZTRACtest_quotas (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16172025-04-30T06:38:14.545ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16182025-04-30T06:38:14.545ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16192025-04-30T06:38:14.545ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16202025-04-30T06:38:14.545ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16212025-04-30T06:38:14.550ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16222025-04-30T06:38:14.564ZDEBGtest_quotas (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16232025-04-30T06:38:14.564ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16242025-04-30T06:38:14.564ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16252025-04-30T06:38:14.564ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16262025-04-30T06:38:14.564ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16272025-04-30T06:38:14.564ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16282025-04-30T06:38:14.567ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16292025-04-30T06:38:14.574ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16302025-04-30T06:38:14.581ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16312025-04-30T06:38:14.581ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16322025-04-30T06:38:14.581ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16332025-04-30T06:38:14.581ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16342025-04-30T06:38:14.582ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16352025-04-30T06:38:14.592ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362025-04-30T06:38:14.612ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16372025-04-30T06:38:14.641ZINFOtest_quotas (InternalApi): Created recovery silo
16382025-04-30T06:38:14.644ZINFOtest_quotas (InternalApi): Created recovery user
16392025-04-30T06:38:14.651ZINFOtest_quotas (InternalApi): Created recovery user's password
16402025-04-30T06:38:14.651ZTRACtest_quotas (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 4b7296ad-2735-4550-bbf6-733aebbb59e4, lookup_type: ById(4b7296ad-2735-4550-bbf6-733aebbb59e4) }
16412025-04-30T06:38:14.651ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 4b7296ad-2735-4550-bbf6-733aebbb59e4 resource_type = Silo
16422025-04-30T06:38:14.651ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16432025-04-30T06:38:14.651ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16442025-04-30T06:38:14.652ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16452025-04-30T06:38:14.658ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16462025-04-30T06:38:14.659ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16472025-04-30T06:38:14.659ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16482025-04-30T06:38:14.660ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16492025-04-30T06:38:14.662ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-04-30T06:38:14.666ZDEBGtest_quotas (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 4b7296ad-2735-4550-bbf6-733aebbb59e4, lookup_type: ById(4b7296ad-2735-4550-bbf6-733aebbb59e4) } result = Ok(())
16512025-04-30T06:38:14.667ZDEBGtest_quotas (InternalApi): Generated role assignment queries
16522025-04-30T06:38:14.676ZINFOtest_quotas (InternalApi): Granted Silo privileges
16532025-04-30T06:38:14.676ZTRACtest_quotas (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16542025-04-30T06:38:14.676ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16552025-04-30T06:38:14.676ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16562025-04-30T06:38:14.677ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16572025-04-30T06:38:14.677ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16582025-04-30T06:38:14.681ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-04-30T06:38:14.685ZDEBGtest_quotas (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16602025-04-30T06:38:14.698ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16612025-04-30T06:38:14.707ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16622025-04-30T06:38:14.708ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16632025-04-30T06:38:14.711ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16642025-04-30T06:38:14.711ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16652025-04-30T06:38:14.713ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16662025-04-30T06:38:14.717ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16672025-04-30T06:38:14.719ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16682025-04-30T06:38:14.720ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16692025-04-30T06:38:14.726ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16702025-04-30T06:38:14.726ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16712025-04-30T06:38:14.726ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16722025-04-30T06:38:14.726ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16732025-04-30T06:38:14.728ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16742025-04-30T06:38:14.744ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16752025-04-30T06:38:14.744ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16762025-04-30T06:38:14.744ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16772025-04-30T06:38:14.745ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16782025-04-30T06:38:14.755ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16792025-04-30T06:38:14.755ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16802025-04-30T06:38:14.755ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16812025-04-30T06:38:14.756ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16822025-04-30T06:38:14.758ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-04-30T06:38:14.767ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
16842025-04-30T06:38:14.768ZTRACtest_quotas (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
16852025-04-30T06:38:14.768ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16862025-04-30T06:38:14.768ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16872025-04-30T06:38:14.768ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16882025-04-30T06:38:14.768ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16892025-04-30T06:38:14.769ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16902025-04-30T06:38:14.784ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16912025-04-30T06:38:14.784ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16922025-04-30T06:38:14.784ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16932025-04-30T06:38:14.785ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16942025-04-30T06:38:14.788ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16952025-04-30T06:38:14.788ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16962025-04-30T06:38:14.788ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
16972025-04-30T06:38:14.789ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16982025-04-30T06:38:14.793ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16992025-04-30T06:38:14.799ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17002025-04-30T06:38:14.799ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17012025-04-30T06:38:14.799ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17022025-04-30T06:38:14.800ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17032025-04-30T06:38:14.805ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17042025-04-30T06:38:14.806ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17052025-04-30T06:38:14.808ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17062025-04-30T06:38:14.812ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17072025-04-30T06:38:14.812ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17082025-04-30T06:38:14.812ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17092025-04-30T06:38:14.829ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17102025-04-30T06:38:14.829ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17112025-04-30T06:38:14.830ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17122025-04-30T06:38:14.836ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
17132025-04-30T06:38:14.836ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17142025-04-30T06:38:14.836ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17152025-04-30T06:38:14.836ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17162025-04-30T06:38:14.837ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17172025-04-30T06:38:14.839ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17182025-04-30T06:38:14.839ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17192025-04-30T06:38:14.839ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17202025-04-30T06:38:14.841ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17212025-04-30T06:38:14.845ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17222025-04-30T06:38:14.845ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17232025-04-30T06:38:14.845ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17242025-04-30T06:38:14.846ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17252025-04-30T06:38:14.854ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262025-04-30T06:38:14.867ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17272025-04-30T06:38:14.867ZTRACtest_quotas (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
17282025-04-30T06:38:14.867ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17292025-04-30T06:38:14.867ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17302025-04-30T06:38:14.869ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17312025-04-30T06:38:14.873ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17322025-04-30T06:38:14.873ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17332025-04-30T06:38:14.874ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17342025-04-30T06:38:14.877ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17352025-04-30T06:38:14.878ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17362025-04-30T06:38:14.879ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17372025-04-30T06:38:14.898ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17382025-04-30T06:38:14.901ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17392025-04-30T06:38:14.903ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17402025-04-30T06:38:14.925ZTRACtest_quotas (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17412025-04-30T06:38:14.932ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17422025-04-30T06:38:14.934ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17432025-04-30T06:38:14.934ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17442025-04-30T06:38:14.934ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17452025-04-30T06:38:14.939ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17462025-04-30T06:38:14.939ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17472025-04-30T06:38:14.939ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17482025-04-30T06:38:14.940ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17492025-04-30T06:38:14.950ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17502025-04-30T06:38:14.950ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17512025-04-30T06:38:14.950ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17522025-04-30T06:38:14.950ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17532025-04-30T06:38:14.958ZINFOtest_quotas (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = c920987f-1d0e-4cd0-91b6-107a19a5a911
17542025-04-30T06:38:14.958ZDEBGtest_quotas (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = c920987f-1d0e-4cd0-91b6-107a19a5a911 time_started = 2025-04-30 06:38:13.277606 UTC
17552025-04-30T06:38:14.958ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 5.303857009s iteration = 2 status = {"collection_id":"c920987f-1d0e-4cd0-91b6-107a19a5a911","time_done":"2025-04-30 06:38:14.375973 UTC","time_started":"2025-04-30 06:38:13.277606 UTC"}
17562025-04-30T06:38:14.959ZDEBGtest_quotas (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
17572025-04-30T06:38:14.959ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 50.25\u{b5}s iteration = 2 status = {"error":"task disabled"}
17582025-04-30T06:38:14.959ZDEBGtest_quotas (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
17592025-04-30T06:38:14.959ZWARNtest_quotas (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
17602025-04-30T06:38:14.959ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 16.382\u{b5}s iteration = 2 status = {"error":"no blueprint"}
17612025-04-30T06:38:14.959ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17622025-04-30T06:38:14.967ZDEBGtest_quotas (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17632025-04-30T06:38:14.967ZTRACtest_quotas (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
17642025-04-30T06:38:14.967ZTRACtest_quotas (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17652025-04-30T06:38:14.967ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17662025-04-30T06:38:14.967ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17672025-04-30T06:38:14.967ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17682025-04-30T06:38:14.968ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17692025-04-30T06:38:14.979ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17702025-04-30T06:38:14.980ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17712025-04-30T06:38:14.980ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17722025-04-30T06:38:14.981ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17732025-04-30T06:38:14.988ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17742025-04-30T06:38:14.988ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:38:14.990ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:38:14.991ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:38:14.998ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17782025-04-30T06:38:15.012ZDEBGtest_quotas (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17792025-04-30T06:38:15.012ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17802025-04-30T06:38:15.012ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17812025-04-30T06:38:15.013ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17822025-04-30T06:38:15.019ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17832025-04-30T06:38:15.019ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17842025-04-30T06:38:15.019ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17852025-04-30T06:38:15.023ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17862025-04-30T06:38:15.023ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17872025-04-30T06:38:15.023ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17882025-04-30T06:38:15.029ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17892025-04-30T06:38:15.029ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17902025-04-30T06:38:15.031ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17912025-04-30T06:38:15.036ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17922025-04-30T06:38:15.036ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17932025-04-30T06:38:15.037ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17942025-04-30T06:38:15.043ZTRACtest_quotas (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17952025-04-30T06:38:15.043ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17962025-04-30T06:38:15.043ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17972025-04-30T06:38:15.043ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
17982025-04-30T06:38:15.044ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17992025-04-30T06:38:15.051ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18002025-04-30T06:38:15.051ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18012025-04-30T06:38:15.052ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
18022025-04-30T06:38:15.052ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18032025-04-30T06:38:15.054ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18042025-04-30T06:38:15.055ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18052025-04-30T06:38:15.055ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
18062025-04-30T06:38:15.056ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18072025-04-30T06:38:15.058ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18082025-04-30T06:38:15.067ZDEBGtest_quotas (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18092025-04-30T06:38:15.067ZTRACtest_quotas (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
18102025-04-30T06:38:15.067ZTRACtest_quotas (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
18112025-04-30T06:38:15.067ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
18122025-04-30T06:38:15.067ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18132025-04-30T06:38:15.067ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
18142025-04-30T06:38:15.068ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18152025-04-30T06:38:15.070ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18162025-04-30T06:38:15.070ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18172025-04-30T06:38:15.070ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
18182025-04-30T06:38:15.071ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18192025-04-30T06:38:15.073ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18202025-04-30T06:38:15.073ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18212025-04-30T06:38:15.073ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
18222025-04-30T06:38:15.074ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18232025-04-30T06:38:15.077ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18242025-04-30T06:38:15.084ZDEBGtest_quotas (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18252025-04-30T06:38:15.084ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18262025-04-30T06:38:15.085ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
18272025-04-30T06:38:15.085ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18282025-04-30T06:38:15.096ZDEBGtest_quotas (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 47f3e88c-4a89-474f-acdf-58e50c206885, kind: Service { id: 503d917a-5d80-4b21-9607-0ae96af250a8 }, name: Name("external-dns-503d917a-5d80-4b21-9607-0ae96af250a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 553b3b5c-2cc8-402e-b852-d2d96d27d777, kind: Service { id: 4678dd50-d1e0-4228-8533-c4090342ff58 }, name: Name("nexus-4678dd50-d1e0-4228-8533-c4090342ff58"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
18292025-04-30T06:38:15.096ZTRACtest_quotas (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
18302025-04-30T06:38:15.097ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18312025-04-30T06:38:15.097ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18322025-04-30T06:38:15.097ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
18332025-04-30T06:38:15.097ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18342025-04-30T06:38:15.100ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18352025-04-30T06:38:15.105ZDEBGtest_quotas (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18362025-04-30T06:38:15.105ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18372025-04-30T06:38:15.105ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
18382025-04-30T06:38:15.105ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18392025-04-30T06:38:15.107ZTRACtest_quotas (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:38:14.685180Z, allowed_ips: None }
18402025-04-30T06:38:15.107ZDEBGtest_quotas (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
18412025-04-30T06:38:15.108ZDEBGtest_quotas (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 47f3e88c-4a89-474f-acdf-58e50c206885, kind: Service { id: 503d917a-5d80-4b21-9607-0ae96af250a8 }, name: Name("external-dns-503d917a-5d80-4b21-9607-0ae96af250a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 553b3b5c-2cc8-402e-b852-d2d96d27d777, kind: Service { id: 4678dd50-d1e0-4228-8533-c4090342ff58 }, name: Name("nexus-4678dd50-d1e0-4228-8533-c4090342ff58"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
18422025-04-30T06:38:15.108ZDEBGtest_quotas (nexus): resolved 2 rules for sleds
18432025-04-30T06:38:15.207ZDEBGtest_quotas (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:38:08.111203Z, time_modified: 2025-04-30T06:38:08.111203Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(55655), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(44150) }]
18442025-04-30T06:38:15.207ZDEBGtest_quotas (nexus): sending firewall rules to sled agents
18452025-04-30T06:38:15.207ZTRACtest_quotas (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18462025-04-30T06:38:15.207ZDEBGtest_quotas (InstanceAllocator): roles roles = RoleSet { roles: {} }
18472025-04-30T06:38:15.208ZDEBGtest_quotas (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18482025-04-30T06:38:15.213ZTRACtest_quotas (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
18492025-04-30T06:38:15.213ZTRACtest_quotas (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18502025-04-30T06:38:15.213ZTRACtest_quotas (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18512025-04-30T06:38:15.213ZDEBGtest_quotas (InstanceAllocator): roles roles = RoleSet { roles: {} }
18522025-04-30T06:38:15.213ZDEBGtest_quotas (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18532025-04-30T06:38:15.216ZDEBGtest_quotas (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
18542025-04-30T06:38:15.220ZDEBGtest_quotas (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
18552025-04-30T06:38:15.263ZDEBGtest_quotas (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:55655/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18562025-04-30T06:38:15.264ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:58203 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
18572025-04-30T06:38:15.265ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = PUT remote_addr = [::1]:58203 req_id = 590ef587-1f3d-4c69-8f3f-9fea4c78425c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18582025-04-30T06:38:15.267ZINFOtest_quotas (dropshot): request completed latency_us = 1409 local_addr = [::1]:55655 method = PUT remote_addr = [::1]:58203 req_id = 590ef587-1f3d-4c69-8f3f-9fea4c78425c response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18592025-04-30T06:38:15.267ZDEBGtest_quotas (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:55655/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "590ef587-1f3d-4c69-8f3f-9fea4c78425c", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
18602025-04-30T06:38:15.269ZINFOtest_quotas (nexus): updated firewall rules on 1 sleds
18612025-04-30T06:38:15.269ZTRACtest_quotas (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
18622025-04-30T06:38:15.269ZDEBGtest_quotas (ServiceBalancer): roles roles = RoleSet { roles: {} }
18632025-04-30T06:38:15.270ZDEBGtest_quotas (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
18642025-04-30T06:38:15.270ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18652025-04-30T06:38:15.270ZDEBGtest_quotas (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18662025-04-30T06:38:15.270ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18672025-04-30T06:38:15.270ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-04-30T06:38:15.271ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-04-30T06:38:15.271ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18702025-04-30T06:38:15.271ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18712025-04-30T06:38:15.271ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18722025-04-30T06:38:15.271ZDEBGtest_quotas (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18732025-04-30T06:38:15.271ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
18742025-04-30T06:38:15.271ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18752025-04-30T06:38:15.271ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18762025-04-30T06:38:15.271ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18772025-04-30T06:38:15.271ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18782025-04-30T06:38:15.272ZDEBGtest_quotas (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18792025-04-30T06:38:15.272ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
18802025-04-30T06:38:15.272ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18812025-04-30T06:38:15.272ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
18822025-04-30T06:38:15.272ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18832025-04-30T06:38:15.272ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18842025-04-30T06:38:15.272ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18852025-04-30T06:38:15.272ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
18862025-04-30T06:38:15.272ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872025-04-30T06:38:15.272ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
18882025-04-30T06:38:15.272ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18892025-04-30T06:38:15.273ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
18902025-04-30T06:38:15.274ZTRACtest_quotas (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 06 82 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:63493 req_id = 3b8e4e8a-56f2-48dc-a67e-dc813c85bb04
18912025-04-30T06:38:15.274ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1666,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63493 req_id = 3b8e4e8a-56f2-48dc-a67e-dc813c85bb04
18922025-04-30T06:38:15.274ZDEBGtest_quotas (store): query key key = _nameservice._tcp
18932025-04-30T06:38:15.274ZDEBGtest_quotas (store): query key key = 4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host
18942025-04-30T06:38:15.275ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63493 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 57119, target: Name("4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host.control-plane.oxide.internal") })) }] req_id = 3b8e4e8a-56f2-48dc-a67e-dc813c85bb04
18952025-04-30T06:38:15.275ZTRACtest_quotas (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: cc a2 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57814 req_id = fdbc8a3e-004d-49d9-9325-e9bedd29e6d7
18962025-04-30T06:38:15.275ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52386,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57814 req_id = fdbc8a3e-004d-49d9-9325-e9bedd29e6d7
18972025-04-30T06:38:15.275ZDEBGtest_quotas (store): query key key = _external-dns._tcp
18982025-04-30T06:38:15.275ZDEBGtest_quotas (store): query key key = 503d917a-5d80-4b21-9607-0ae96af250a8.host
18992025-04-30T06:38:15.275ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("503d917a-5d80-4b21-9607-0ae96af250a8.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57814 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36358, target: Name("503d917a-5d80-4b21-9607-0ae96af250a8.host.control-plane.oxide.internal") })) }] req_id = fdbc8a3e-004d-49d9-9325-e9bedd29e6d7
19002025-04-30T06:38:15.276ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 57119, target: Name("4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3960, tv_nsec: 75236163 } })
19012025-04-30T06:38:15.276ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
19022025-04-30T06:38:15.277ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36358, target: Name("503d917a-5d80-4b21-9607-0ae96af250a8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("503d917a-5d80-4b21-9607-0ae96af250a8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3960, tv_nsec: 75989143 } })
19032025-04-30T06:38:15.277ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
19042025-04-30T06:38:15.277ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19052025-04-30T06:38:15.277ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19062025-04-30T06:38:15.277ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19072025-04-30T06:38:15.277ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19082025-04-30T06:38:15.279ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19092025-04-30T06:38:15.279ZTRACtest_quotas (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
19102025-04-30T06:38:15.279ZTRACtest_quotas (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19112025-04-30T06:38:15.279ZTRACtest_quotas (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
19122025-04-30T06:38:15.279ZDEBGtest_quotas (ServiceBalancer): roles roles = RoleSet { roles: {} }
19132025-04-30T06:38:15.280ZDEBGtest_quotas (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
19142025-04-30T06:38:15.281ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19152025-04-30T06:38:15.281ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19162025-04-30T06:38:15.281ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19172025-04-30T06:38:15.281ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19182025-04-30T06:38:15.282ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19192025-04-30T06:38:15.282ZTRACtest_quotas (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 98 90 01 00 00 01 00 00 00 00 00 01 24 34 37 31 ............$471\n0010: 38 66 34 35 62 2d 31 35 64 64 2d 34 65 63 39 2d 8f45b-15dd-4ec9-\n0020: 39 63 64 33 2d 64 62 36 62 39 39 61 36 61 61 34 9cd3-db6b99a6aa4\n0030: 66 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 f.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:50448 req_id = 0850f035-c667-4356-85bf-4b6b22e1f000
19202025-04-30T06:38:15.283ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39056,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 52,\n 55,\n 49,\n 56,\n 102,\n 52,\n 53,\n 98,\n 45,\n 49,\n 53,\n 100,\n 100,\n 45,\n 52,\n 101,\n 99,\n 57,\n 45,\n 57,\n 99,\n 100,\n 51,\n 45,\n 100,\n 98,\n 54,\n 98,\n 57,\n 57,\n 97,\n 54,\n 97,\n 97,\n 52,\n 102,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50448 req_id = 0850f035-c667-4356-85bf-4b6b22e1f000
19212025-04-30T06:38:15.283ZDEBGtest_quotas (store): query key key = 4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host
19222025-04-30T06:38:15.283ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50448 query = LowerQuery { name: LowerName(Name("4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host.control-plane.oxide.internal.")), original: Query { name: Name("4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("4718f45b-15dd-4ec9-9cd3-db6b99a6aa4f.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0850f035-c667-4356-85bf-4b6b22e1f000
19232025-04-30T06:38:15.283ZTRACtest_quotas (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: bf 16 01 00 00 01 00 00 00 00 00 01 24 35 30 33 ............$503\n0010: 64 39 31 37 61 2d 35 64 38 30 2d 34 62 32 31 2d d917a-5d80-4b21-\n0020: 39 36 30 37 2d 30 61 65 39 36 61 66 32 35 30 61 9607-0ae96af250a\n0030: 38 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 8.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:51796 req_id = 2d78a82c-bd30-4310-8549-19de207cde08
19242025-04-30T06:38:15.283ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48918,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("503d917a-5d80-4b21-9607-0ae96af250a8.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("503d917a-5d80-4b21-9607-0ae96af250a8.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 53,\n 48,\n 51,\n 100,\n 57,\n 49,\n 55,\n 97,\n 45,\n 53,\n 100,\n 56,\n 48,\n 45,\n 52,\n 98,\n 50,\n 49,\n 45,\n 57,\n 54,\n 48,\n 55,\n 45,\n 48,\n 97,\n 101,\n 57,\n 54,\n 97,\n 102,\n 50,\n 53,\n 48,\n 97,\n 56,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51796 req_id = 2d78a82c-bd30-4310-8549-19de207cde08
19252025-04-30T06:38:15.284ZDEBGtest_quotas (store): query key key = 503d917a-5d80-4b21-9607-0ae96af250a8.host
19262025-04-30T06:38:15.284ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51796 query = LowerQuery { name: LowerName(Name("503d917a-5d80-4b21-9607-0ae96af250a8.host.control-plane.oxide.internal.")), original: Query { name: Name("503d917a-5d80-4b21-9607-0ae96af250a8.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("503d917a-5d80-4b21-9607-0ae96af250a8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2d78a82c-bd30-4310-8549-19de207cde08
19272025-04-30T06:38:15.286ZINFOtest_quotas (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:57119] } background_task = dns_servers_internal dns_group = internal
19282025-04-30T06:38:15.286ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 15.097521ms iteration = 2 status = {"addresses":["[::1]:57119"]}
19292025-04-30T06:38:15.286ZINFOtest_quotas (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:36358] } background_task = dns_servers_external dns_group = external
19302025-04-30T06:38:15.286ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 14.649178ms iteration = 2 status = {"addresses":["[::1]:36358"]}
19312025-04-30T06:38:15.286ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19322025-04-30T06:38:15.286ZWARNtest_quotas (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19332025-04-30T06:38:15.286ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 27.194\u{b5}s iteration = 2 status = {"error":"no config"}
19342025-04-30T06:38:15.286ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19352025-04-30T06:38:15.286ZWARNtest_quotas (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19362025-04-30T06:38:15.286ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 11.293\u{b5}s iteration = 2 status = {"error":"no config"}
19372025-04-30T06:38:15.286ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382025-04-30T06:38:15.293ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19392025-04-30T06:38:15.293ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19402025-04-30T06:38:15.293ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19412025-04-30T06:38:15.293ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19422025-04-30T06:38:15.295ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19432025-04-30T06:38:15.300ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19442025-04-30T06:38:15.300ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19452025-04-30T06:38:15.300ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19462025-04-30T06:38:15.301ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19472025-04-30T06:38:15.301ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19482025-04-30T06:38:15.308ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19492025-04-30T06:38:15.310ZDEBGtest_quotas (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19502025-04-30T06:38:15.311ZDEBGtest_quotas (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19512025-04-30T06:38:15.311ZDEBGtest_quotas (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19522025-04-30T06:38:15.311ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19532025-04-30T06:38:15.311ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19542025-04-30T06:38:15.311ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19552025-04-30T06:38:15.311ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19562025-04-30T06:38:15.311ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19572025-04-30T06:38:15.312ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19582025-04-30T06:38:15.317ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19592025-04-30T06:38:15.318ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
19602025-04-30T06:38:15.318ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19612025-04-30T06:38:15.318ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19622025-04-30T06:38:15.318ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19632025-04-30T06:38:15.319ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19642025-04-30T06:38:15.320ZDEBGtest_quotas (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19652025-04-30T06:38:15.320ZDEBGtest_quotas (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19662025-04-30T06:38:15.320ZDEBGtest_quotas (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19672025-04-30T06:38:15.320ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19682025-04-30T06:38:15.321ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19692025-04-30T06:38:15.321ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19702025-04-30T06:38:15.321ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19712025-04-30T06:38:15.321ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19722025-04-30T06:38:15.324ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
19732025-04-30T06:38:15.324ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19742025-04-30T06:38:15.325ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19752025-04-30T06:38:15.332ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19762025-04-30T06:38:15.332ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19772025-04-30T06:38:15.332ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19782025-04-30T06:38:15.333ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19792025-04-30T06:38:15.333ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19802025-04-30T06:38:15.340ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19812025-04-30T06:38:15.340ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19822025-04-30T06:38:15.340ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19832025-04-30T06:38:15.341ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19842025-04-30T06:38:15.342ZDEBGtest_quotas (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19852025-04-30T06:38:15.348ZDEBGtest_quotas (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
19862025-04-30T06:38:15.348ZINFOtest_quotas (nexus): Rack initialized
19872025-04-30T06:38:15.348ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19882025-04-30T06:38:15.348ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
19892025-04-30T06:38:15.348ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19902025-04-30T06:38:15.349ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19912025-04-30T06:38:15.367ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19922025-04-30T06:38:15.367ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19932025-04-30T06:38:15.367ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19942025-04-30T06:38:15.367ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19952025-04-30T06:38:15.369ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e7 c4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49973 req_id = 9345aced-9e0a-4746-a338-ed16e5f8b179
19962025-04-30T06:38:15.369ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59332,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49973 req_id = 9345aced-9e0a-4746-a338-ed16e5f8b179
19972025-04-30T06:38:15.369ZDEBGtest_quotas (store): query key key = _mgs._tcp
19982025-04-30T06:38:15.369ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19992025-04-30T06:38:15.369ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49973 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9345aced-9e0a-4746-a338-ed16e5f8b179
20002025-04-30T06:38:15.370ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3960, tv_nsec: 169438906 } })
20012025-04-30T06:38:15.370ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
20022025-04-30T06:38:15.372ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 82 ce 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64866 req_id = e54391a2-488f-4de3-97bf-fda2fe82e92a
20032025-04-30T06:38:15.372ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33486,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64866 req_id = e54391a2-488f-4de3-97bf-fda2fe82e92a
20042025-04-30T06:38:15.374ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
20052025-04-30T06:38:15.375ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64866 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e54391a2-488f-4de3-97bf-fda2fe82e92a
20062025-04-30T06:38:15.375ZDEBGtest_quotas (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20072025-04-30T06:38:15.375ZDEBGtest_quotas (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 416f1b32-0425-418f-8531-8f3d38acb21c dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
20082025-04-30T06:38:15.375ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
20092025-04-30T06:38:15.375ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20102025-04-30T06:38:15.375ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20112025-04-30T06:38:15.375ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20122025-04-30T06:38:15.376ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20132025-04-30T06:38:15.377ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20142025-04-30T06:38:15.377ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
20152025-04-30T06:38:15.377ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20162025-04-30T06:38:15.432ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
20172025-04-30T06:38:15.434ZDEBGtest_quotas (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
20182025-04-30T06:38:15.434ZDEBGtest_quotas (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 06733288-7bcd-49f4-b2ab-1637e4ebde66 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
20192025-04-30T06:38:15.434ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
20202025-04-30T06:38:15.434ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20212025-04-30T06:38:15.434ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20222025-04-30T06:38:15.434ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20232025-04-30T06:38:15.435ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20242025-04-30T06:38:15.435ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
20252025-04-30T06:38:15.435ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20262025-04-30T06:38:15.435ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20272025-04-30T06:38:15.435ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20282025-04-30T06:38:15.436ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20292025-04-30T06:38:15.437ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20302025-04-30T06:38:15.443ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20312025-04-30T06:38:15.443ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20322025-04-30T06:38:15.443ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20332025-04-30T06:38:15.449ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20342025-04-30T06:38:15.450ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20352025-04-30T06:38:15.466ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20362025-04-30T06:38:15.466ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20372025-04-30T06:38:15.466ZDEBGtest_quotas (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20382025-04-30T06:38:15.469ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20392025-04-30T06:38:15.470ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20402025-04-30T06:38:15.495ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20412025-04-30T06:38:15.495ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20422025-04-30T06:38:15.495ZDEBGtest_quotas (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20432025-04-30T06:38:15.495ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20442025-04-30T06:38:15.496ZDEBGtest_quotas (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 416f1b32-0425-418f-8531-8f3d38acb21c dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
20452025-04-30T06:38:15.496ZDEBGtest_quotas (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20462025-04-30T06:38:15.496ZINFOtest_quotas (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20472025-04-30T06:38:15.497ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 225.115208ms iteration = 2 status = {"generation":2}
20482025-04-30T06:38:15.497ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20492025-04-30T06:38:15.562ZDEBGtest_quotas (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:36358 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:36358] } uri = http://[::1]:36358/config
20502025-04-30T06:38:15.563ZDEBGtest_quotas (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 06733288-7bcd-49f4-b2ab-1637e4ebde66 dns_zone_name = control-plane.oxide.internal found_so_far = 21 version = 1
20512025-04-30T06:38:15.563ZDEBGtest_quotas (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20522025-04-30T06:38:15.563ZINFOtest_quotas (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20532025-04-30T06:38:15.563ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 293.37925ms iteration = 2 status = {"generation":1}
20542025-04-30T06:38:15.563ZDEBGtest_quotas (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20552025-04-30T06:38:15.618ZDEBGtest_quotas (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:57119 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:57119] } uri = http://[::1]:57119/config
20562025-04-30T06:38:15.619ZINFOtest_quotas (http): accepted connection kind = dns local_addr = [::1]:57119 remote_addr = [::1]:59793
20572025-04-30T06:38:15.620ZINFOtest_quotas (http): accepted connection kind = dns local_addr = [::1]:36358 remote_addr = [::1]:48987
20582025-04-30T06:38:15.625ZTRACtest_quotas (http): incoming request kind = dns local_addr = [::1]:57119 method = PUT remote_addr = [::1]:59793 req_id = aa132423-f66d-455a-8443-efe1dbb3a70c uri = /config
20592025-04-30T06:38:15.626ZTRACtest_quotas (http): incoming request kind = dns local_addr = [::1]:36358 method = PUT remote_addr = [::1]:48987 req_id = fd797528-d3d7-4292-96ca-5b00a51dd825 uri = /config
20602025-04-30T06:38:15.626ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7b dc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c {............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64693 req_id = f918a527-b11b-4f2d-8875-2e222989dd14
20612025-04-30T06:38:15.627ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31708,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64693 req_id = f918a527-b11b-4f2d-8875-2e222989dd14
20622025-04-30T06:38:15.627ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
20632025-04-30T06:38:15.627ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31708, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64693 req_id = f918a527-b11b-4f2d-8875-2e222989dd14
20642025-04-30T06:38:15.628ZINFOtest_quotas (store): attempting generation update new_generation = 1 req_id = aa132423-f66d-455a-8443-efe1dbb3a70c
20652025-04-30T06:38:15.628ZINFOtest_quotas (store): updated generation new_generation = 1 req_id = aa132423-f66d-455a-8443-efe1dbb3a70c
20662025-04-30T06:38:15.628ZINFOtest_quotas (store): attempting generation update new_generation = 2 req_id = fd797528-d3d7-4292-96ca-5b00a51dd825
20672025-04-30T06:38:15.628ZINFOtest_quotas (store): pruning trees for generations newer than 0
20682025-04-30T06:38:15.629ZDEBGtest_quotas (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20692025-04-30T06:38:15.629ZDEBGtest_quotas (store): updating current config
20702025-04-30T06:38:15.630ZDEBGtest_quotas (store): flushing default tree
20712025-04-30T06:38:15.630ZINFOtest_quotas (store): pruning trees for generations older than 2 keep = 3
20722025-04-30T06:38:15.630ZINFOtest_quotas (store): updated generation new_generation = 2 req_id = fd797528-d3d7-4292-96ca-5b00a51dd825
20732025-04-30T06:38:15.630ZINFOtest_quotas (http): request completed kind = dns latency_us = 5021 local_addr = [::1]:57119 method = PUT remote_addr = [::1]:59793 req_id = aa132423-f66d-455a-8443-efe1dbb3a70c response_code = 204 uri = /config
20742025-04-30T06:38:15.630ZINFOtest_quotas (http): request completed kind = dns latency_us = 4860 local_addr = [::1]:36358 method = PUT remote_addr = [::1]:48987 req_id = fd797528-d3d7-4292-96ca-5b00a51dd825 response_code = 204 uri = /config
20752025-04-30T06:38:15.631ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20762025-04-30T06:38:15.631ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
20772025-04-30T06:38:15.632ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20782025-04-30T06:38:15.632ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
20792025-04-30T06:38:15.632ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
20802025-04-30T06:38:15.633ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
20812025-04-30T06:38:15.633ZDEBGtest_quotas (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:36358 generation = 2 result = Ok(Response { url: "http://[::1]:36358/config", status: 204, headers: {"x-request-id": "fd797528-d3d7-4292-96ca-5b00a51dd825", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} }) servers = DnsServersList { addresses: [[::1]:36358] }
20822025-04-30T06:38:15.633ZINFOtest_quotas (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:36358 generation = 2 servers = DnsServersList { addresses: [[::1]:36358] }
20832025-04-30T06:38:15.635ZINFOtest_quotas (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:36358] }
20842025-04-30T06:38:15.635ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 138.887865ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:36358":{"Ok":null}}}
20852025-04-30T06:38:15.635ZDEBGtest_quotas (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:57119 generation = 1 result = Ok(Response { url: "http://[::1]:57119/config", status: 204, headers: {"x-request-id": "aa132423-f66d-455a-8443-efe1dbb3a70c", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} }) servers = DnsServersList { addresses: [[::1]:57119] }
20862025-04-30T06:38:15.636ZINFOtest_quotas (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:57119 generation = 1 servers = DnsServersList { addresses: [[::1]:57119] }
20872025-04-30T06:38:15.637ZINFOtest_quotas (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:57119] }
20882025-04-30T06:38:15.637ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 73.355025ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:57119":{"Ok":null}}}
20892025-04-30T06:38:15.640ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "90c7176e-2e97-43b7-ad1d-76500c9a3a41", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
20902025-04-30T06:38:15.641ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
20912025-04-30T06:38:15.641ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20922025-04-30T06:38:15.641ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20932025-04-30T06:38:15.641ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20942025-04-30T06:38:15.641ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
20952025-04-30T06:38:15.642ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20962025-04-30T06:38:15.645ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53d03c9b-4574-4a82-80db-1bb7baeedb31", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
20972025-04-30T06:38:15.645ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
20982025-04-30T06:38:15.645ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20992025-04-30T06:38:15.645ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21002025-04-30T06:38:15.646ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21012025-04-30T06:38:15.646ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21022025-04-30T06:38:15.649ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21032025-04-30T06:38:15.649ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21042025-04-30T06:38:15.649ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21052025-04-30T06:38:15.649ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21062025-04-30T06:38:15.651ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b5e9391-66af-4786-8bcf-4677be5b85db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21072025-04-30T06:38:15.651ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
21082025-04-30T06:38:15.652ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21092025-04-30T06:38:15.661ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21102025-04-30T06:38:15.662ZTRACtest_quotas (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21112025-04-30T06:38:15.662ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21122025-04-30T06:38:15.662ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21132025-04-30T06:38:15.662ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21142025-04-30T06:38:15.662ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21152025-04-30T06:38:15.663ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21162025-04-30T06:38:15.663ZINFOtest_quotas (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:38:06.280214Z, time_modified: 2025-04-30T06:38:06.280214Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 4b7296ad-2735-4550-bbf6-733aebbb59e4, db_silo: Silo { identity: SiloIdentity { id: 4b7296ad-2735-4550-bbf6-733aebbb59e4, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:38:14.484295Z, time_modified: 2025-04-30T06:38:14.484295Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: 4b7296ad-2735-4550-bbf6-733aebbb59e4, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 4b7296ad-2735-4550-bbf6-733aebbb59e4, db_silo: Silo { identity: SiloIdentity { id: 4b7296ad-2735-4550-bbf6-733aebbb59e4, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:38:14.484295Z, time_modified: 2025-04-30T06:38:14.484295Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
21172025-04-30T06:38:15.664ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 391.946538ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"4b7296ad-2735-4550-bbf6-733aebbb59e4","tls_certs":[]}},"default_endpoint":{"silo_id":"4b7296ad-2735-4550-bbf6-733aebbb59e4","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo 4b7296ad-2735-4550-bbf6-733aebbb59e4 with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
21182025-04-30T06:38:15.665ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21192025-04-30T06:38:15.665ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21202025-04-30T06:38:15.665ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21212025-04-30T06:38:15.666ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21222025-04-30T06:38:15.668ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21232025-04-30T06:38:15.668ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21242025-04-30T06:38:15.668ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21252025-04-30T06:38:15.669ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21262025-04-30T06:38:15.669ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4b47c63-345a-440f-a785-8221330cbb4d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21272025-04-30T06:38:15.670ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
21282025-04-30T06:38:15.672ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21292025-04-30T06:38:15.677ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21302025-04-30T06:38:15.678ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21312025-04-30T06:38:15.678ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21322025-04-30T06:38:15.678ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21332025-04-30T06:38:15.680ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3d1b4fa-b667-410b-8e37-5a29d5f219f6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21342025-04-30T06:38:15.680ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
21352025-04-30T06:38:15.683ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21362025-04-30T06:38:15.684ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21372025-04-30T06:38:15.684ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21382025-04-30T06:38:15.686ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e19f4d5-cdc1-4301-b8ee-6b0bdd000f46", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21392025-04-30T06:38:15.686ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
21402025-04-30T06:38:15.687ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21412025-04-30T06:38:15.687ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21422025-04-30T06:38:15.687ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21432025-04-30T06:38:15.690ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a94a48e9-b8b7-4958-aaa0-460e8728e3ee", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21442025-04-30T06:38:15.690ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
21452025-04-30T06:38:15.690ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21462025-04-30T06:38:15.691ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21472025-04-30T06:38:15.691ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21482025-04-30T06:38:15.694ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ce6c000-3271-457c-aebb-b0e94b3844ed", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21492025-04-30T06:38:15.694ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
21502025-04-30T06:38:15.694ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21512025-04-30T06:38:15.695ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21522025-04-30T06:38:15.695ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21532025-04-30T06:38:15.695ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21542025-04-30T06:38:15.696ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21552025-04-30T06:38:15.697ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38873f44-4f26-49ec-b630-dc4b61811d2a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21562025-04-30T06:38:15.697ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
21572025-04-30T06:38:15.699ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1085c93-8a44-48e8-8b21-71e9b4f94dfd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21582025-04-30T06:38:15.699ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
21592025-04-30T06:38:15.699ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21602025-04-30T06:38:15.700ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21612025-04-30T06:38:15.700ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21622025-04-30T06:38:15.701ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21632025-04-30T06:38:15.702ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9b9d165-f497-4370-9a71-e0d07f320e14", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21642025-04-30T06:38:15.703ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
21652025-04-30T06:38:15.704ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21662025-04-30T06:38:15.704ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21672025-04-30T06:38:15.704ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21682025-04-30T06:38:15.705ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21692025-04-30T06:38:15.707ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4449814e-3b11-447a-b80a-4aeaf624f0f8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21702025-04-30T06:38:15.707ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
21712025-04-30T06:38:15.708ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21722025-04-30T06:38:15.716ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21732025-04-30T06:38:15.717ZTRACtest_quotas (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21742025-04-30T06:38:15.717ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21752025-04-30T06:38:15.717ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21762025-04-30T06:38:15.718ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21772025-04-30T06:38:15.719ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d26d471-fd2e-4c96-8471-8cf34c86f6dd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21782025-04-30T06:38:15.720ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
21792025-04-30T06:38:15.721ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21802025-04-30T06:38:15.721ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21812025-04-30T06:38:15.721ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21822025-04-30T06:38:15.723ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afed15a7-a0b2-41c6-a4bd-fd9edea22195", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21832025-04-30T06:38:15.724ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
21842025-04-30T06:38:15.727ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d4938ae-367b-4fa5-b106-b6337bb5b6c5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21852025-04-30T06:38:15.728ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
21862025-04-30T06:38:15.729ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21872025-04-30T06:38:15.729ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21882025-04-30T06:38:15.730ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21892025-04-30T06:38:15.733ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21902025-04-30T06:38:15.733ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21912025-04-30T06:38:15.734ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21922025-04-30T06:38:15.736ZTRACtest_quotas (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
21932025-04-30T06:38:15.736ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21942025-04-30T06:38:15.736ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21952025-04-30T06:38:15.736ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
21962025-04-30T06:38:15.736ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21972025-04-30T06:38:15.737ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a73d825-b3c1-4fb2-8a97-73c9dd254822", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
21982025-04-30T06:38:15.737ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
21992025-04-30T06:38:15.739ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22002025-04-30T06:38:15.739ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22012025-04-30T06:38:15.739ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22022025-04-30T06:38:15.740ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22032025-04-30T06:38:15.741ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "abeda6e9-1452-4215-bd95-e60f18046a2d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22042025-04-30T06:38:15.741ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
22052025-04-30T06:38:15.743ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22062025-04-30T06:38:15.744ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22072025-04-30T06:38:15.744ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22082025-04-30T06:38:15.745ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22092025-04-30T06:38:15.746ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "753182ba-70ce-4184-8aa6-89910fe8a3b1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22102025-04-30T06:38:15.746ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
22112025-04-30T06:38:15.748ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22122025-04-30T06:38:15.755ZDEBGtest_quotas (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22132025-04-30T06:38:15.755ZTRACtest_quotas (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22142025-04-30T06:38:15.755ZTRACtest_quotas (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22152025-04-30T06:38:15.755ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22162025-04-30T06:38:15.755ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22172025-04-30T06:38:15.755ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22182025-04-30T06:38:15.756ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22192025-04-30T06:38:15.756ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "533e8574-2df0-41bf-8062-940421efed43", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22202025-04-30T06:38:15.757ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
22212025-04-30T06:38:15.758ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b9ef5de-a118-4025-a68f-4c15cf55f958", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22222025-04-30T06:38:15.758ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
22232025-04-30T06:38:15.759ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22242025-04-30T06:38:15.759ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22252025-04-30T06:38:15.759ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22262025-04-30T06:38:15.759ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22272025-04-30T06:38:15.760ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "932fbc11-0667-4f6c-8789-ed34f7b9c784", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22282025-04-30T06:38:15.760ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
22292025-04-30T06:38:15.763ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97fe790c-99f3-4c42-960f-8168e4c8fcc8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22302025-04-30T06:38:15.763ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
22312025-04-30T06:38:15.764ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22322025-04-30T06:38:15.764ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22332025-04-30T06:38:15.764ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22342025-04-30T06:38:15.764ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22352025-04-30T06:38:15.766ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c989690-ecca-4319-bfff-1eb625f12fb7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22362025-04-30T06:38:15.766ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
22372025-04-30T06:38:15.767ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a235b93-9214-4875-aa88-07a884af0107", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22382025-04-30T06:38:15.768ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
22392025-04-30T06:38:15.768ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22402025-04-30T06:38:15.775ZDEBGtest_quotas (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22412025-04-30T06:38:15.775ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22422025-04-30T06:38:15.775ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22432025-04-30T06:38:15.776ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22442025-04-30T06:38:15.782ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39cea5c1-69c2-43be-9336-505968b4f74e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22452025-04-30T06:38:15.783ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
22462025-04-30T06:38:15.784ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22472025-04-30T06:38:15.784ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22482025-04-30T06:38:15.785ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22492025-04-30T06:38:15.790ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "daaecbfa-8a55-4168-9ea8-a6918b792ddb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22502025-04-30T06:38:15.790ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
22512025-04-30T06:38:15.790ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a614548e-2a71-4076-97cb-73ea3ab7df8b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22522025-04-30T06:38:15.790ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
22532025-04-30T06:38:15.792ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51a5bff3-737c-483c-bfe5-be9376fd3eff", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22542025-04-30T06:38:15.792ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
22552025-04-30T06:38:15.794ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88bcaf1e-aa4b-4a4c-bd07-24329973cb29", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22562025-04-30T06:38:15.794ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
22572025-04-30T06:38:15.795ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22582025-04-30T06:38:15.796ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22592025-04-30T06:38:15.797ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22602025-04-30T06:38:15.797ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "033293bc-3653-4bf3-a951-a93ca2b6709a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22612025-04-30T06:38:15.797ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
22622025-04-30T06:38:15.799ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e94c7f6c-1aee-4328-91cc-d3cb562f0592", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22632025-04-30T06:38:15.799ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
22642025-04-30T06:38:15.799ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22652025-04-30T06:38:15.799ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22662025-04-30T06:38:15.800ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22672025-04-30T06:38:15.801ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96a89cab-6f55-4505-9901-b98dbd172c79", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22682025-04-30T06:38:15.801ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
22692025-04-30T06:38:15.802ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ffad512-f6f6-4c15-b197-c5a6ab802005", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22702025-04-30T06:38:15.802ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
22712025-04-30T06:38:15.803ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22722025-04-30T06:38:15.803ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22732025-04-30T06:38:15.803ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22742025-04-30T06:38:15.804ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ccc5259-f3d6-4377-b707-35ca965a4be9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22752025-04-30T06:38:15.805ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
22762025-04-30T06:38:15.806ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca4d4242-a79d-4aaa-ac6d-f6b5dbfc3ea6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22772025-04-30T06:38:15.806ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
22782025-04-30T06:38:15.807ZTRACtest_quotas (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22792025-04-30T06:38:15.807ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22802025-04-30T06:38:15.807ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22812025-04-30T06:38:15.807ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22822025-04-30T06:38:15.807ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22832025-04-30T06:38:15.809ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22842025-04-30T06:38:15.809ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22852025-04-30T06:38:15.809ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22862025-04-30T06:38:15.810ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22872025-04-30T06:38:15.811ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64627d21-050a-4e7e-a5b0-f8137ae68bf4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
22882025-04-30T06:38:15.811ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
22892025-04-30T06:38:15.814ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22902025-04-30T06:38:15.814ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22912025-04-30T06:38:15.814ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
22922025-04-30T06:38:15.814ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22932025-04-30T06:38:15.817ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22942025-04-30T06:38:15.824ZDEBGtest_quotas (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22952025-04-30T06:38:15.824ZTRACtest_quotas (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22962025-04-30T06:38:15.824ZTRACtest_quotas (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22972025-04-30T06:38:15.824ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22982025-04-30T06:38:15.824ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22992025-04-30T06:38:15.824ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
23002025-04-30T06:38:15.825ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23012025-04-30T06:38:15.825ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9e782ec-4ede-4d70-9497-f38e93cca3af", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
23022025-04-30T06:38:15.825ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
23032025-04-30T06:38:15.828ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
23042025-04-30T06:38:15.828ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23052025-04-30T06:38:15.828ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
23062025-04-30T06:38:15.828ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23072025-04-30T06:38:15.831ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23082025-04-30T06:38:15.831ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23092025-04-30T06:38:15.831ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
23102025-04-30T06:38:15.831ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23112025-04-30T06:38:15.833ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64b302b9-f616-4c61-8c92-a4219f0f66f2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:15 GMT"} })
23122025-04-30T06:38:15.833ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
23132025-04-30T06:38:15.835ZTRACtest_quotas (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 88 cb 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:59516 req_id = 26c64835-2c19-4e47-a783-b1e9b0125e9c
23142025-04-30T06:38:15.835ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35019,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59516 req_id = 26c64835-2c19-4e47-a783-b1e9b0125e9c
23152025-04-30T06:38:15.835ZDEBGtest_quotas (store): query key key = _crucible-pantry._tcp
23162025-04-30T06:38:15.835ZDEBGtest_quotas (store): query key key = f0fa6243-1db8-401f-9a4d-243fcac463b4.host
23172025-04-30T06:38:15.835ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("f0fa6243-1db8-401f-9a4d-243fcac463b4.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59516 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40067, target: Name("f0fa6243-1db8-401f-9a4d-243fcac463b4.host.control-plane.oxide.internal") })) }] req_id = 26c64835-2c19-4e47-a783-b1e9b0125e9c
23182025-04-30T06:38:15.836ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23192025-04-30T06:38:15.844ZDEBGtest_quotas (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
23202025-04-30T06:38:15.844ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23212025-04-30T06:38:15.844ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
23222025-04-30T06:38:15.844ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23232025-04-30T06:38:15.844ZTRACtest_quotas (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: d0 c9 01 00 00 01 00 00 00 00 00 01 24 66 30 66 ............$f0f\n0010: 61 36 32 34 33 2d 31 64 62 38 2d 34 30 31 66 2d a6243-1db8-401f-\n0020: 39 61 34 64 2d 32 34 33 66 63 61 63 34 36 33 62 9a4d-243fcac463b\n0030: 34 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 4.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:52370 req_id = 070a4ff2-0df5-4fdc-95da-006747f5b8f9
23242025-04-30T06:38:15.844ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53449,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("f0fa6243-1db8-401f-9a4d-243fcac463b4.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("f0fa6243-1db8-401f-9a4d-243fcac463b4.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 102,\n 48,\n 102,\n 97,\n 54,\n 50,\n 52,\n 51,\n 45,\n 49,\n 100,\n 98,\n 56,\n 45,\n 52,\n 48,\n 49,\n 102,\n 45,\n 57,\n 97,\n 52,\n 100,\n 45,\n 50,\n 52,\n 51,\n 102,\n 99,\n 97,\n 99,\n 52,\n 54,\n 51,\n 98,\n 52,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52370 req_id = 070a4ff2-0df5-4fdc-95da-006747f5b8f9
23252025-04-30T06:38:15.844ZDEBGtest_quotas (store): query key key = f0fa6243-1db8-401f-9a4d-243fcac463b4.host
23262025-04-30T06:38:15.844ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52370 query = LowerQuery { name: LowerName(Name("f0fa6243-1db8-401f-9a4d-243fcac463b4.host.control-plane.oxide.internal.")), original: Query { name: Name("f0fa6243-1db8-401f-9a4d-243fcac463b4.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("f0fa6243-1db8-401f-9a4d-243fcac463b4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 070a4ff2-0df5-4fdc-95da-006747f5b8f9
23272025-04-30T06:38:15.897ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:57055 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23282025-04-30T06:38:15.899ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:57055 req_id = c58f48c8-d654-4dbd-ac35-be3fe943a241 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23292025-04-30T06:38:15.899ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 529 local_addr = [::1]:40067 method = GET remote_addr = [::1]:57055 req_id = c58f48c8-d654-4dbd-ac35-be3fe943a241 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23302025-04-30T06:38:16.324ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:61545 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23312025-04-30T06:38:16.324ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:33576 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23322025-04-30T06:38:16.325ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:36707 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23332025-04-30T06:38:16.325ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:36090 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23342025-04-30T06:38:16.325ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:50650 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23352025-04-30T06:38:16.325ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:61389 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23362025-04-30T06:38:16.325ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:35926 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23372025-04-30T06:38:16.327ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:61545 req_id = 078fc708-6127-4d26-a8c4-122bd6c3a362 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23382025-04-30T06:38:16.327ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:33576 req_id = eef09c5b-0c73-4970-89e9-4cf4d13c7d37 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23392025-04-30T06:38:16.327ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:36707 req_id = 64cd66ba-78dd-487b-8eaa-1176555c6b4f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23402025-04-30T06:38:16.327ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:36090 req_id = ce7c35ab-1b68-4afe-a77a-66c86cc642d5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23412025-04-30T06:38:16.328ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:50650 req_id = 7bdb758e-b073-4685-b5c4-25b2451fa083 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23422025-04-30T06:38:16.328ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:61389 req_id = 8616a3f1-910f-4903-b5c0-057df02328e6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23432025-04-30T06:38:16.328ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:35926 req_id = 70a52f2b-debf-4449-8321-399f0ec1e9f7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23442025-04-30T06:38:16.328ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1460 local_addr = [::1]:40067 method = GET remote_addr = [::1]:61545 req_id = 078fc708-6127-4d26-a8c4-122bd6c3a362 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23452025-04-30T06:38:16.328ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1360 local_addr = [::1]:40067 method = GET remote_addr = [::1]:33576 req_id = eef09c5b-0c73-4970-89e9-4cf4d13c7d37 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23462025-04-30T06:38:16.329ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1346 local_addr = [::1]:40067 method = GET remote_addr = [::1]:36707 req_id = 64cd66ba-78dd-487b-8eaa-1176555c6b4f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23472025-04-30T06:38:16.329ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1373 local_addr = [::1]:40067 method = GET remote_addr = [::1]:36090 req_id = ce7c35ab-1b68-4afe-a77a-66c86cc642d5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23482025-04-30T06:38:16.329ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1205 local_addr = [::1]:40067 method = GET remote_addr = [::1]:50650 req_id = 7bdb758e-b073-4685-b5c4-25b2451fa083 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23492025-04-30T06:38:16.329ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1119 local_addr = [::1]:40067 method = GET remote_addr = [::1]:61389 req_id = 8616a3f1-910f-4903-b5c0-057df02328e6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23502025-04-30T06:38:16.329ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1035 local_addr = [::1]:40067 method = GET remote_addr = [::1]:35926 req_id = 70a52f2b-debf-4449-8321-399f0ec1e9f7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
23512025-04-30T06:38:16.330ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ee5be02-4d96-47ab-a250-d6897e41430b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:16 GMT"} })
23522025-04-30T06:38:16.330ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
23532025-04-30T06:38:16.332ZDEBGtest_quotas (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 47f3e88c-4a89-474f-acdf-58e50c206885, kind: Service { id: 503d917a-5d80-4b21-9607-0ae96af250a8 }, name: Name("external-dns-503d917a-5d80-4b21-9607-0ae96af250a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 553b3b5c-2cc8-402e-b852-d2d96d27d777, kind: Service { id: 4678dd50-d1e0-4228-8533-c4090342ff58 }, name: Name("nexus-4678dd50-d1e0-4228-8533-c4090342ff58"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
23542025-04-30T06:38:16.332ZTRACtest_quotas (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
23552025-04-30T06:38:16.332ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23562025-04-30T06:38:16.332ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23572025-04-30T06:38:16.332ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
23582025-04-30T06:38:16.333ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23592025-04-30T06:38:16.337ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23602025-04-30T06:38:16.345ZDEBGtest_quotas (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
23612025-04-30T06:38:16.345ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23622025-04-30T06:38:16.345ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
23632025-04-30T06:38:16.345ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23642025-04-30T06:38:16.351ZTRACtest_quotas (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:38:14.685180Z, allowed_ips: None }
23652025-04-30T06:38:16.351ZDEBGtest_quotas (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
23662025-04-30T06:38:16.351ZDEBGtest_quotas (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 47f3e88c-4a89-474f-acdf-58e50c206885, kind: Service { id: 503d917a-5d80-4b21-9607-0ae96af250a8 }, name: Name("external-dns-503d917a-5d80-4b21-9607-0ae96af250a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 553b3b5c-2cc8-402e-b852-d2d96d27d777, kind: Service { id: 4678dd50-d1e0-4228-8533-c4090342ff58 }, name: Name("nexus-4678dd50-d1e0-4228-8533-c4090342ff58"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
23672025-04-30T06:38:16.352ZDEBGtest_quotas (InternalApi): resolved 2 rules for sleds
23682025-04-30T06:38:16.393ZDEBGtest_quotas (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:38:08.111203Z, time_modified: 2025-04-30T06:38:08.111203Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(55655), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(44150) }]
23692025-04-30T06:38:16.393ZDEBGtest_quotas (InternalApi): sending firewall rules to sled agents
23702025-04-30T06:38:16.393ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23712025-04-30T06:38:16.393ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
23722025-04-30T06:38:16.393ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23732025-04-30T06:38:16.399ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
23742025-04-30T06:38:16.399ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23752025-04-30T06:38:16.399ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23762025-04-30T06:38:16.399ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
23772025-04-30T06:38:16.400ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23782025-04-30T06:38:16.406ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23792025-04-30T06:38:16.414ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
23802025-04-30T06:38:16.460ZDEBGtest_quotas (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:55655/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23812025-04-30T06:38:16.461ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7df4322-284d-4935-875e-56d18a44f6b7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:16 GMT"} })
23822025-04-30T06:38:16.461ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
23832025-04-30T06:38:16.462ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:37873 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23842025-04-30T06:38:16.464ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = PUT remote_addr = [::1]:37873 req_id = cc60b7ae-7d82-4e36-8a58-830e294bef8a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23852025-04-30T06:38:16.465ZINFOtest_quotas (dropshot): request completed latency_us = 1282 local_addr = [::1]:55655 method = PUT remote_addr = [::1]:37873 req_id = cc60b7ae-7d82-4e36-8a58-830e294bef8a response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23862025-04-30T06:38:16.466ZDEBGtest_quotas (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:55655/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "cc60b7ae-7d82-4e36-8a58-830e294bef8a", "date": "Wed, 30 Apr 2025 06:38:16 GMT"} })
23872025-04-30T06:38:16.466ZINFOtest_quotas (InternalApi): updated firewall rules on 1 sleds
23882025-04-30T06:38:16.466ZINFOtest_quotas (nexus): plumbed initial IP allowlist
23892025-04-30T06:38:16.532ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = / versions = all
23902025-04-30T06:38:16.532ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /access versions = all
23912025-04-30T06:38:16.532ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /assets/{path} versions = all
23922025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /device/auth versions = all
23932025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /device/confirm versions = all
23942025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /device/success versions = all
23952025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /device/token versions = all
23962025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /device/verify versions = all
23972025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /experimental/v1/probes versions = all
23982025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /experimental/v1/probes versions = all
23992025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /experimental/v1/probes/{probe} versions = all
24002025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /experimental/v1/probes/{probe} versions = all
24012025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /experimental/v1/system/support-bundles versions = all
24022025-04-30T06:38:16.533ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /experimental/v1/system/support-bundles versions = all
24032025-04-30T06:38:16.534ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
24042025-04-30T06:38:16.534ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
24052025-04-30T06:38:16.534ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
24062025-04-30T06:38:16.534ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
24072025-04-30T06:38:16.534ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
24082025-04-30T06:38:16.534ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
24092025-04-30T06:38:16.534ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
24102025-04-30T06:38:16.534ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /images versions = all
24112025-04-30T06:38:16.534ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /login versions = all
24122025-04-30T06:38:16.535ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /login/{silo_name}/local versions = all
24132025-04-30T06:38:16.535ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
24142025-04-30T06:38:16.535ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
24152025-04-30T06:38:16.535ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
24162025-04-30T06:38:16.535ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /lookup/{path} versions = all
24172025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /projects/{path} versions = all
24182025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /projects-new versions = all
24192025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /settings/{path} versions = all
24202025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /system/{path} versions = all
24212025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /utilization versions = all
24222025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/affinity-groups versions = all
24232025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/affinity-groups versions = all
24242025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
24252025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
24262025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
24272025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
24282025-04-30T06:38:16.536ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24292025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24302025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24312025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/anti-affinity-groups versions = all
24322025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/anti-affinity-groups versions = all
24332025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24342025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24352025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24362025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
24372025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24382025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24392025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24402025-04-30T06:38:16.537ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/certificates versions = all
24412025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/certificates versions = all
24422025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/certificates/{certificate} versions = all
24432025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/certificates/{certificate} versions = all
24442025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/disks versions = all
24452025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/disks versions = all
24462025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/disks/{disk} versions = all
24472025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/disks/{disk} versions = all
24482025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/disks/{disk}/bulk-write versions = all
24492025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
24502025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
24512025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/disks/{disk}/finalize versions = all
24522025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
24532025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/floating-ips versions = all
24542025-04-30T06:38:16.538ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/floating-ips versions = all
24552025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
24562025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/floating-ips/{floating_ip} versions = all
24572025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
24582025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
24592025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
24602025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/groups versions = all
24612025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/groups/{group_id} versions = all
24622025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/images versions = all
24632025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/images versions = all
24642025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/images/{image} versions = all
24652025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/images/{image} versions = all
24662025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/images/{image}/demote versions = all
24672025-04-30T06:38:16.539ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/images/{image}/promote versions = all
24682025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/instances versions = all
24692025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/instances versions = all
24702025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/instances/{instance} versions = all
24712025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/instances/{instance} versions = all
24722025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/instances/{instance} versions = all
24732025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
24742025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
24752025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/instances/{instance}/disks versions = all
24762025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/instances/{instance}/disks/attach versions = all
24772025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/instances/{instance}/disks/detach versions = all
24782025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/instances/{instance}/external-ips versions = all
24792025-04-30T06:38:16.540ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24802025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24812025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/instances/{instance}/reboot versions = all
24822025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/instances/{instance}/serial-console versions = all
24832025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
24842025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
24852025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/instances/{instance}/start versions = all
24862025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/instances/{instance}/stop versions = all
24872025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/internet-gateway-ip-addresses versions = all
24882025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/internet-gateway-ip-addresses versions = all
24892025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
24902025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/internet-gateway-ip-pools versions = all
24912025-04-30T06:38:16.541ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/internet-gateway-ip-pools versions = all
24922025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
24932025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/internet-gateways versions = all
24942025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/internet-gateways versions = all
24952025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
24962025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/internet-gateways/{gateway} versions = all
24972025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/ip-pools versions = all
24982025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/ip-pools/{pool} versions = all
24992025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/login/{silo_name}/local versions = all
25002025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/logout versions = all
25012025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/me versions = all
25022025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/me/groups versions = all
25032025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/me/ssh-keys versions = all
25042025-04-30T06:38:16.542ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/me/ssh-keys versions = all
25052025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
25062025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
25072025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/metrics/{metric_name} versions = all
25082025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/network-interfaces versions = all
25092025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/network-interfaces versions = all
25102025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/network-interfaces/{interface} versions = all
25112025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/network-interfaces/{interface} versions = all
25122025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/network-interfaces/{interface} versions = all
25132025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/ping versions = all
25142025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/policy versions = all
25152025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/policy versions = all
25162025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/projects versions = all
25172025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/projects versions = all
25182025-04-30T06:38:16.543ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/projects/{project} versions = all
25192025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/projects/{project} versions = all
25202025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/projects/{project} versions = all
25212025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/projects/{project}/policy versions = all
25222025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/projects/{project}/policy versions = all
25232025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/snapshots versions = all
25242025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/snapshots versions = all
25252025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/snapshots/{snapshot} versions = all
25262025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/snapshots/{snapshot} versions = all
25272025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/disks versions = all
25282025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
25292025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
25302025-04-30T06:38:16.544ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/racks versions = all
25312025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
25322025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/sleds versions = all
25332025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/hardware/sleds versions = all
25342025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
25352025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
25362025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
25372025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
25382025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
25392025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/switch-port versions = all
25402025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25412025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25422025-04-30T06:38:16.545ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
25432025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
25442025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
25452025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/switches versions = all
25462025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
25472025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/identity-providers versions = all
25482025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/identity-providers/local/users versions = all
25492025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
25502025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
25512025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/identity-providers/saml versions = all
25522025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
25532025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/ip-pools versions = all
25542025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/ip-pools versions = all
25552025-04-30T06:38:16.546ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
25562025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/ip-pools/{pool} versions = all
25572025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/system/ip-pools/{pool} versions = all
25582025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
25592025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
25602025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
25612025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
25622025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
25632025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25642025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25652025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
25662025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/ip-pools-service versions = all
25672025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/ip-pools-service/ranges versions = all
25682025-04-30T06:38:16.547ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
25692025-04-30T06:38:16.548ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
25702025-04-30T06:38:16.549ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/metrics/{metric_name} versions = all
25712025-04-30T06:38:16.549ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/address-lot versions = all
25722025-04-30T06:38:16.549ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/networking/address-lot versions = all
25732025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
25742025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
25752025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/allow-list versions = all
25762025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/system/networking/allow-list versions = all
25772025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/networking/bfd-disable versions = all
25782025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/networking/bfd-enable versions = all
25792025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/bfd-status versions = all
25802025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/system/networking/bgp versions = all
25812025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/bgp versions = all
25822025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/networking/bgp versions = all
25832025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/bgp-announce-set versions = all
25842025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
25852025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
25862025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
25872025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/bgp-exported versions = all
25882025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/bgp-message-history versions = all
25892025-04-30T06:38:16.550ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
25902025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/bgp-status versions = all
25912025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/loopback-address versions = all
25922025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/networking/loopback-address versions = all
25932025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
25942025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
25952025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/switch-port-settings versions = all
25962025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/networking/switch-port-settings versions = all
25972025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
25982025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/policy versions = all
25992025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/system/policy versions = all
26002025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/roles versions = all
26012025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/roles/{role_name} versions = all
26022025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/silo-quotas versions = all
26032025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/silos versions = all
26042025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/silos versions = all
26052025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/system/silos/{silo} versions = all
26062025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/silos/{silo} versions = all
26072025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
26082025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/silos/{silo}/policy versions = all
26092025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/system/silos/{silo}/policy versions = all
26102025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/silos/{silo}/quotas versions = all
26112025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
26122025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/system/timeseries/query versions = all
26132025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/timeseries/schemas versions = all
26142025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/system/update/repository versions = all
26152025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/update/repository/{system_version} versions = all
26162025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/update/target-release versions = all
26172025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/system/update/target-release versions = all
26182025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/users versions = all
26192025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/users/{user_id} versions = all
26202025-04-30T06:38:16.551ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/users-builtin versions = all
26212025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/users-builtin/{user} versions = all
26222025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/utilization/silos versions = all
26232025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/system/utilization/silos/{silo} versions = all
26242025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/timeseries/query versions = all
26252025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/users versions = all
26262025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/utilization versions = all
26272025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/vpc-firewall-rules versions = all
26282025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/vpc-firewall-rules versions = all
26292025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/vpc-router-routes versions = all
26302025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/vpc-router-routes versions = all
26312025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
26322025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/vpc-router-routes/{route} versions = all
26332025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/vpc-router-routes/{route} versions = all
26342025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/vpc-routers versions = all
26352025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/vpc-routers versions = all
26362025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/vpc-routers/{router} versions = all
26372025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/vpc-routers/{router} versions = all
26382025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/vpc-routers/{router} versions = all
26392025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/vpc-subnets versions = all
26402025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/vpc-subnets versions = all
26412025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
26422025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/vpc-subnets/{subnet} versions = all
26432025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
26442025-04-30T06:38:16.552ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
26452025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/vpcs versions = all
26462025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/vpcs versions = all
26472025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/vpcs/{vpc} versions = all
26482025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/vpcs/{vpc} versions = all
26492025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/vpcs/{vpc} versions = all
26502025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/webhooks/deliveries versions = all
26512025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
26522025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/webhooks/event-classes versions = all
26532025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/webhooks/receivers versions = all
26542025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/webhooks/receivers versions = all
26552025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
26562025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
26572025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
26582025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
26592025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
26602025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
26612025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = GET path = /v1/webhooks/secrets versions = all
26622025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = POST path = /v1/webhooks/secrets versions = all
26632025-04-30T06:38:16.553ZDEBGtest_quotas (dropshot_external): registered endpoint local_addr = 127.0.0.1:45178 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
26642025-04-30T06:38:16.553ZINFOtest_quotas (dropshot_external): listening local_addr = 127.0.0.1:45178
26652025-04-30T06:38:16.555ZDEBGtest_quotas (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:45178
26662025-04-30T06:38:16.628ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = / versions = all
26672025-04-30T06:38:16.628ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /access versions = all
26682025-04-30T06:38:16.628ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /assets/{path} versions = all
26692025-04-30T06:38:16.628ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /device/auth versions = all
26702025-04-30T06:38:16.628ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /device/confirm versions = all
26712025-04-30T06:38:16.628ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /device/success versions = all
26722025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /device/token versions = all
26732025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /device/verify versions = all
26742025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /experimental/v1/probes versions = all
26752025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /experimental/v1/probes versions = all
26762025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /experimental/v1/probes/{probe} versions = all
26772025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /experimental/v1/probes/{probe} versions = all
26782025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /experimental/v1/system/support-bundles versions = all
26792025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /experimental/v1/system/support-bundles versions = all
26802025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26812025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26822025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26832025-04-30T06:38:16.629ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26842025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26852025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26862025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
26872025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /images versions = all
26882025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /login versions = all
26892025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /login/{silo_name}/local versions = all
26902025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
26912025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
26922025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
26932025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /lookup/{path} versions = all
26942025-04-30T06:38:16.630ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /projects/{path} versions = all
26952025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /projects-new versions = all
26962025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /settings/{path} versions = all
26972025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /system/{path} versions = all
26982025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /utilization versions = all
26992025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/affinity-groups versions = all
27002025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/affinity-groups versions = all
27012025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
27022025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
27032025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
27042025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
27052025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27062025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27072025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27082025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/anti-affinity-groups versions = all
27092025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/anti-affinity-groups versions = all
27102025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27112025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27122025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27132025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
27142025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27152025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27162025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27172025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/certificates versions = all
27182025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/certificates versions = all
27192025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/certificates/{certificate} versions = all
27202025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/certificates/{certificate} versions = all
27212025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/disks versions = all
27222025-04-30T06:38:16.631ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/disks versions = all
27232025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/disks/{disk} versions = all
27242025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/disks/{disk} versions = all
27252025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/disks/{disk}/bulk-write versions = all
27262025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
27272025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
27282025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/disks/{disk}/finalize versions = all
27292025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
27302025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/floating-ips versions = all
27312025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/floating-ips versions = all
27322025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
27332025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/floating-ips/{floating_ip} versions = all
27342025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
27352025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
27362025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
27372025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/groups versions = all
27382025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/groups/{group_id} versions = all
27392025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/images versions = all
27402025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/images versions = all
27412025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/images/{image} versions = all
27422025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/images/{image} versions = all
27432025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/images/{image}/demote versions = all
27442025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/images/{image}/promote versions = all
27452025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/instances versions = all
27462025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/instances versions = all
27472025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/instances/{instance} versions = all
27482025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/instances/{instance} versions = all
27492025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/instances/{instance} versions = all
27502025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
27512025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
27522025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/instances/{instance}/disks versions = all
27532025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/instances/{instance}/disks/attach versions = all
27542025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/instances/{instance}/disks/detach versions = all
27552025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/instances/{instance}/external-ips versions = all
27562025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27572025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27582025-04-30T06:38:16.632ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/instances/{instance}/reboot versions = all
27592025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/instances/{instance}/serial-console versions = all
27602025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
27612025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
27622025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/instances/{instance}/start versions = all
27632025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/instances/{instance}/stop versions = all
27642025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/internet-gateway-ip-addresses versions = all
27652025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/internet-gateway-ip-addresses versions = all
27662025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
27672025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/internet-gateway-ip-pools versions = all
27682025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/internet-gateway-ip-pools versions = all
27692025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
27702025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/internet-gateways versions = all
27712025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/internet-gateways versions = all
27722025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
27732025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/internet-gateways/{gateway} versions = all
27742025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/ip-pools versions = all
27752025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/ip-pools/{pool} versions = all
27762025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/login/{silo_name}/local versions = all
27772025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/logout versions = all
27782025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/me versions = all
27792025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/me/groups versions = all
27802025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/me/ssh-keys versions = all
27812025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/me/ssh-keys versions = all
27822025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
27832025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
27842025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/metrics/{metric_name} versions = all
27852025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/network-interfaces versions = all
27862025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/network-interfaces versions = all
27872025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/network-interfaces/{interface} versions = all
27882025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/network-interfaces/{interface} versions = all
27892025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/network-interfaces/{interface} versions = all
27902025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/ping versions = all
27912025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/policy versions = all
27922025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/policy versions = all
27932025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/projects versions = all
27942025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/projects versions = all
27952025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/projects/{project} versions = all
27962025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/projects/{project} versions = all
27972025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/projects/{project} versions = all
27982025-04-30T06:38:16.633ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/projects/{project}/policy versions = all
27992025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/projects/{project}/policy versions = all
28002025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/snapshots versions = all
28012025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/snapshots versions = all
28022025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/snapshots/{snapshot} versions = all
28032025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/snapshots/{snapshot} versions = all
28042025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/disks versions = all
28052025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
28062025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
28072025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/racks versions = all
28082025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
28092025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/sleds versions = all
28102025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/hardware/sleds versions = all
28112025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
28122025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
28132025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
28142025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
28152025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
28162025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/switch-port versions = all
28172025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
28182025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
28192025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
28202025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
28212025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
28222025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/switches versions = all
28232025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
28242025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/identity-providers versions = all
28252025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/identity-providers/local/users versions = all
28262025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
28272025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
28282025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/identity-providers/saml versions = all
28292025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
28302025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/ip-pools versions = all
28312025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/ip-pools versions = all
28322025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
28332025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/ip-pools/{pool} versions = all
28342025-04-30T06:38:16.634ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/system/ip-pools/{pool} versions = all
28352025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
28362025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
28372025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
28382025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
28392025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
28402025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28412025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28422025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
28432025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/ip-pools-service versions = all
28442025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/ip-pools-service/ranges versions = all
28452025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
28462025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
28472025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/metrics/{metric_name} versions = all
28482025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/address-lot versions = all
28492025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/networking/address-lot versions = all
28502025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
28512025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
28522025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/allow-list versions = all
28532025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/system/networking/allow-list versions = all
28542025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/networking/bfd-disable versions = all
28552025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/networking/bfd-enable versions = all
28562025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/bfd-status versions = all
28572025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/system/networking/bgp versions = all
28582025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/bgp versions = all
28592025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/networking/bgp versions = all
28602025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/bgp-announce-set versions = all
28612025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
28622025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
28632025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
28642025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/bgp-exported versions = all
28652025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/bgp-message-history versions = all
28662025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
28672025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/bgp-status versions = all
28682025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/loopback-address versions = all
28692025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/networking/loopback-address versions = all
28702025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28712025-04-30T06:38:16.635ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28722025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/switch-port-settings versions = all
28732025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/networking/switch-port-settings versions = all
28742025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
28752025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/policy versions = all
28762025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/system/policy versions = all
28772025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/roles versions = all
28782025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/roles/{role_name} versions = all
28792025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/silo-quotas versions = all
28802025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/silos versions = all
28812025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/silos versions = all
28822025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/system/silos/{silo} versions = all
28832025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/silos/{silo} versions = all
28842025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
28852025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/silos/{silo}/policy versions = all
28862025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28872025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/silos/{silo}/quotas versions = all
28882025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
28892025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/system/timeseries/query versions = all
28902025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/timeseries/schemas versions = all
28912025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/system/update/repository versions = all
28922025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/update/repository/{system_version} versions = all
28932025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/update/target-release versions = all
28942025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/system/update/target-release versions = all
28952025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/users versions = all
28962025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/users/{user_id} versions = all
28972025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/users-builtin versions = all
28982025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/users-builtin/{user} versions = all
28992025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/utilization/silos versions = all
29002025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/system/utilization/silos/{silo} versions = all
29012025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/timeseries/query versions = all
29022025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/users versions = all
29032025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/utilization versions = all
29042025-04-30T06:38:16.636ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/vpc-firewall-rules versions = all
29052025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/vpc-firewall-rules versions = all
29062025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/vpc-router-routes versions = all
29072025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/vpc-router-routes versions = all
29082025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
29092025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/vpc-router-routes/{route} versions = all
29102025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/vpc-router-routes/{route} versions = all
29112025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/vpc-routers versions = all
29122025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/vpc-routers versions = all
29132025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/vpc-routers/{router} versions = all
29142025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/vpc-routers/{router} versions = all
29152025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/vpc-routers/{router} versions = all
29162025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/vpc-subnets versions = all
29172025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/vpc-subnets versions = all
29182025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
29192025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/vpc-subnets/{subnet} versions = all
29202025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
29212025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
29222025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/vpcs versions = all
29232025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/vpcs versions = all
29242025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/vpcs/{vpc} versions = all
29252025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/vpcs/{vpc} versions = all
29262025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/vpcs/{vpc} versions = all
29272025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/webhooks/deliveries versions = all
29282025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
29292025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/webhooks/event-classes versions = all
29302025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/webhooks/receivers versions = all
29312025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/webhooks/receivers versions = all
29322025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
29332025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
29342025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
29352025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
29362025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
29372025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
29382025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = GET path = /v1/webhooks/secrets versions = all
29392025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = POST path = /v1/webhooks/secrets versions = all
29402025-04-30T06:38:16.637ZDEBGtest_quotas (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:40003 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
29412025-04-30T06:38:16.637ZINFOtest_quotas (dropshot_external_techport): listening local_addr = 127.0.0.1:40003
29422025-04-30T06:38:16.638ZDEBGtest_quotas (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:40003
29432025-04-30T06:38:16.638ZDEBGtest_quotas (nexus-producer-server): registered DTrace probes
29442025-04-30T06:38:16.638ZDEBGtest_quotas (dropshot): registered endpoint local_addr = 127.0.0.1:48722 method = GET path = /{producer_id} versions = all
29452025-04-30T06:38:16.638ZINFOtest_quotas (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:48722
29462025-04-30T06:38:16.638ZDEBGtest_quotas (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:48722
29472025-04-30T06:38:16.638ZDEBGtest_quotas (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:48722
29482025-04-30T06:38:16.638ZDEBGtest_quotas (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:64801
29492025-04-30T06:38:16.638ZDEBGtest_quotas (nexus-producer-server): starting producer registration task
29502025-04-30T06:38:16.638ZINFOtest_quotas (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:48722 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
29512025-04-30T06:38:16.638ZDEBGtest_quotas (init_with_steps): Running step start_oximeter
29522025-04-30T06:38:16.638ZDEBGtest_quotas: Starting Oximeter
29532025-04-30T06:38:16.638ZDEBGtest_quotas (oximeter): registered DTrace probes
29542025-04-30T06:38:16.638ZINFOtest_quotas (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
29552025-04-30T06:38:16.638ZDEBGtest_quotas (oximeter): creating ClickHouse client
29562025-04-30T06:38:16.638ZDEBGtest_quotas (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
29572025-04-30T06:38:16.638ZDEBGtest_quotas (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
29582025-04-30T06:38:16.638ZDEBGtest_quotas (producer-registration-task): will register with Nexus at 127.0.0.1:64801
29592025-04-30T06:38:16.716ZDEBGtest_quotas (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:64801/metrics/producers
29602025-04-30T06:38:16.717ZINFOtest_quotas (dropshot_internal): accepted connection local_addr = 127.0.0.1:64801 remote_addr = 127.0.0.1:45167
29612025-04-30T06:38:16.719ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:45167 req_id = 3c4bdb39-0f81-4e3c-b123-a37d18b9682d uri = /metrics/producers
29622025-04-30T06:38:16.721ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:45167 req_id = 3c4bdb39-0f81-4e3c-b123-a37d18b9682d resource = Database uri = /metrics/producers
29632025-04-30T06:38:16.721ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:45167 req_id = 3c4bdb39-0f81-4e3c-b123-a37d18b9682d roles = RoleSet { roles: {} } uri = /metrics/producers
29642025-04-30T06:38:16.722ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:45167 req_id = 3c4bdb39-0f81-4e3c-b123-a37d18b9682d resource = Database result = Ok(()) uri = /metrics/producers
29652025-04-30T06:38:16.725ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29662025-04-30T06:38:16.729ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fc9b60e-8c89-49fb-8b81-ff422d8d61bb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:16 GMT"} })
29672025-04-30T06:38:16.729ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
29682025-04-30T06:38:16.730ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14a4121c-bca7-43b9-a6fa-c4ad46c9a11c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:16 GMT"} })
29692025-04-30T06:38:16.730ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
29702025-04-30T06:38:16.731ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d12496da-6718-4da8-a985-3dc77ede13b6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:16 GMT"} })
29712025-04-30T06:38:16.731ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
29722025-04-30T06:38:16.732ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "049a48e3-be6f-4708-bc3d-521cc8737903", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:16 GMT"} })
29732025-04-30T06:38:16.733ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
29742025-04-30T06:38:16.735ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b636c756-70d4-47d3-9078-25d89f788811", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:16 GMT"} })
29752025-04-30T06:38:16.735ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29762025-04-30T06:38:16.735ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29772025-04-30T06:38:16.735ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29782025-04-30T06:38:16.735ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29792025-04-30T06:38:16.736ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29802025-04-30T06:38:16.738ZWARNtest_quotas (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = a24f3fbe-6c58-493e-9445-8078333aa41b
29812025-04-30T06:38:16.738ZDEBGtest_quotas (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
29822025-04-30T06:38:16.738ZINFOtest_quotas (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = a24f3fbe-6c58-493e-9445-8078333aa41b
29832025-04-30T06:38:16.738ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29842025-04-30T06:38:16.739ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29852025-04-30T06:38:16.743ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29862025-04-30T06:38:16.743ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29872025-04-30T06:38:16.743ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29882025-04-30T06:38:16.743ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29892025-04-30T06:38:16.743ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29902025-04-30T06:38:16.744ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29912025-04-30T06:38:16.744ZWARNtest_quotas (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = a24f3fbe-6c58-493e-9445-8078333aa41b
29922025-04-30T06:38:16.744ZINFOtest_quotas (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = a24f3fbe-6c58-493e-9445-8078333aa41b version = 0
29932025-04-30T06:38:16.744ZINFOtest_quotas (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = a24f3fbe-6c58-493e-9445-8078333aa41b
29942025-04-30T06:38:16.744ZDEBGtest_quotas (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
29952025-04-30T06:38:16.744ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = CREATE DATABASE IF NOT EXISTS oximeter
29962025-04-30T06:38:16.746ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
29972025-04-30T06:38:16.754ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29982025-04-30T06:38:16.759ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29992025-04-30T06:38:16.759ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30002025-04-30T06:38:16.759ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30012025-04-30T06:38:16.760ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30022025-04-30T06:38:16.761ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30032025-04-30T06:38:16.765ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30042025-04-30T06:38:16.823ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
30052025-04-30T06:38:16.823ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
30062025-04-30T06:38:16.824ZINFOtest_quotas (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 105132 local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:45167 req_id = 3c4bdb39-0f81-4e3c-b123-a37d18b9682d response_code = 503 uri = /metrics/producers
30072025-04-30T06:38:16.825ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30082025-04-30T06:38:16.825ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:42516 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30092025-04-30T06:38:16.826ZDEBGtest_quotas (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "3c4bdb39-0f81-4e3c-b123-a37d18b9682d", "content-length": "133", "date": "Wed, 30 Apr 2025 06:38:16 GMT"} })
30102025-04-30T06:38:16.829ZWARNtest_quotas (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 190.103098ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"3c4bdb39-0f81-4e3c-b123-a37d18b9682d\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:38:16 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"3c4bdb39-0f81-4e3c-b123-a37d18b9682d\\" }" file = oximeter/producer/src/lib.rs:371
30112025-04-30T06:38:16.829ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30122025-04-30T06:38:16.830ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:42516 req_id = e5f1cfbb-9a2f-4217-9291-fbb8c018857c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
30132025-04-30T06:38:16.831ZINFOtest_quotas (dropshot): request completed latency_us = 549 local_addr = [::1]:55655 method = GET remote_addr = [::1]:42516 req_id = e5f1cfbb-9a2f-4217-9291-fbb8c018857c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
30142025-04-30T06:38:16.831ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5f1cfbb-9a2f-4217-9291-fbb8c018857c", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:38:16 GMT"} })
30152025-04-30T06:38:16.836ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
30162025-04-30T06:38:16.836ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
30172025-04-30T06:38:16.837ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
30182025-04-30T06:38:16.840ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
30192025-04-30T06:38:16.841ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
30202025-04-30T06:38:16.841ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30212025-04-30T06:38:16.841ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30222025-04-30T06:38:16.844ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30232025-04-30T06:38:16.844ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30242025-04-30T06:38:16.846ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30252025-04-30T06:38:16.852ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
30262025-04-30T06:38:16.852ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
30272025-04-30T06:38:16.852ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
30282025-04-30T06:38:16.853ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30292025-04-30T06:38:16.854ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30302025-04-30T06:38:16.858ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30312025-04-30T06:38:16.862ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30322025-04-30T06:38:16.866ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30332025-04-30T06:38:16.869ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30342025-04-30T06:38:16.874ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30352025-04-30T06:38:16.880ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30362025-04-30T06:38:16.884ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30372025-04-30T06:38:16.888ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30382025-04-30T06:38:16.894ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30392025-04-30T06:38:16.902ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30402025-04-30T06:38:16.911ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30412025-04-30T06:38:16.917ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30422025-04-30T06:38:16.927ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30432025-04-30T06:38:16.938ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30442025-04-30T06:38:16.944ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30452025-04-30T06:38:16.950ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30462025-04-30T06:38:16.975ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30472025-04-30T06:38:16.977ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
30482025-04-30T06:38:16.978ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
30492025-04-30T06:38:16.978ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.705478655s iteration = 3 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
30502025-04-30T06:38:16.980ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30512025-04-30T06:38:16.985ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30522025-04-30T06:38:16.992ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30532025-04-30T06:38:17.013ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30542025-04-30T06:38:17.018ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30552025-04-30T06:38:17.022ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30562025-04-30T06:38:17.025ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30572025-04-30T06:38:17.030ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30582025-04-30T06:38:17.034ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30592025-04-30T06:38:17.046ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30602025-04-30T06:38:17.050ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30612025-04-30T06:38:17.056ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30622025-04-30T06:38:17.061ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30632025-04-30T06:38:17.066ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30642025-04-30T06:38:17.071ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30652025-04-30T06:38:17.081ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30662025-04-30T06:38:17.085ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
30672025-04-30T06:38:17.092ZINFOtest_quotas (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = a24f3fbe-6c58-493e-9445-8078333aa41b version = 14
30682025-04-30T06:38:17.092ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = INSERT INTO oximeter.version (*) VALUES (14, now());
30692025-04-30T06:38:17.099ZDEBGtest_quotas (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 457fa665-c3e2-4bad-9627-59cb0e982558
30702025-04-30T06:38:17.100ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:38322 method = GET path = /info versions = all
30712025-04-30T06:38:17.100ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:38322 method = GET path = /producers versions = all
30722025-04-30T06:38:17.100ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:38322 method = DELETE path = /producers/{producer_id} versions = all
30732025-04-30T06:38:17.100ZDEBGtest_quotas (dropshot): registered endpoint local_addr = [::1]:38322 method = GET path = /producers/{producer_id} versions = all
30742025-04-30T06:38:17.100ZINFOtest_quotas (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:38322
30752025-04-30T06:38:17.101ZDEBGtest_quotas (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:38322
30762025-04-30T06:38:17.102ZDEBGtest_quotas (oximeter): registered USDT probes
30772025-04-30T06:38:17.102ZDEBGtest_quotas (oximeter): contacting nexus
30782025-04-30T06:38:17.646ZDEBGtest_quotas (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:64801/metrics/collectors
30792025-04-30T06:38:17.646ZDEBGtest_quotas (producer-registration-task): will register with Nexus at 127.0.0.1:64801
30802025-04-30T06:38:17.720ZDEBGtest_quotas (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:64801/metrics/producers
30812025-04-30T06:38:17.721ZINFOtest_quotas (dropshot_internal): accepted connection local_addr = 127.0.0.1:64801 remote_addr = 127.0.0.1:40669
30822025-04-30T06:38:17.721ZINFOtest_quotas (dropshot_internal): accepted connection local_addr = 127.0.0.1:64801 remote_addr = 127.0.0.1:46130
30832025-04-30T06:38:17.722ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:46130 req_id = a2c84460-b262-4c1a-adbb-610a4215544c uri = /metrics/producers
30842025-04-30T06:38:17.722ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:46130 req_id = a2c84460-b262-4c1a-adbb-610a4215544c resource = Database uri = /metrics/producers
30852025-04-30T06:38:17.723ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:46130 req_id = a2c84460-b262-4c1a-adbb-610a4215544c roles = RoleSet { roles: {} } uri = /metrics/producers
30862025-04-30T06:38:17.723ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:46130 req_id = a2c84460-b262-4c1a-adbb-610a4215544c resource = Database result = Ok(()) uri = /metrics/producers
30872025-04-30T06:38:17.724ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:40669 req_id = 9783bf77-1fc5-4d02-bb78-494d78715b49 uri = /metrics/collectors
30882025-04-30T06:38:17.724ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:40669 req_id = 9783bf77-1fc5-4d02-bb78-494d78715b49 resource = Database uri = /metrics/collectors
30892025-04-30T06:38:17.724ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:40669 req_id = 9783bf77-1fc5-4d02-bb78-494d78715b49 roles = RoleSet { roles: {} } uri = /metrics/collectors
30902025-04-30T06:38:17.724ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:40669 req_id = 9783bf77-1fc5-4d02-bb78-494d78715b49 resource = Database result = Ok(()) uri = /metrics/collectors
30912025-04-30T06:38:17.729ZINFOtest_quotas (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 6817 local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:46130 req_id = a2c84460-b262-4c1a-adbb-610a4215544c response_code = 503 uri = /metrics/producers
30922025-04-30T06:38:17.729ZDEBGtest_quotas (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "a2c84460-b262-4c1a-adbb-610a4215544c", "content-length": "133", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
30932025-04-30T06:38:17.731ZWARNtest_quotas (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 1.092781015s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"a2c84460-b262-4c1a-adbb-610a4215544c\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:38:17 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"a2c84460-b262-4c1a-adbb-610a4215544c\\" }" file = oximeter/producer/src/lib.rs:371
30942025-04-30T06:38:17.732ZINFOtest_quotas (nexus): registered new oximeter metric collection server address = [::1]:38322 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
30952025-04-30T06:38:17.732ZINFOtest_quotas (dropshot_internal): request completed latency_us = 8443 local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:40669 req_id = 9783bf77-1fc5-4d02-bb78-494d78715b49 response_code = 204 uri = /metrics/collectors
30962025-04-30T06:38:17.733ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors", status: 204, headers: {"x-request-id": "9783bf77-1fc5-4d02-bb78-494d78715b49", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
30972025-04-30T06:38:17.733ZINFOtest_quotas (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
30982025-04-30T06:38:17.733ZDEBGtest_quotas (init_with_steps): Running step start_producer_server
30992025-04-30T06:38:17.733ZDEBGtest_quotas: Starting test metric Producer Server
31002025-04-30T06:38:17.736ZDEBGtest_quotas (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:43201 method = GET path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31012025-04-30T06:38:17.736ZDEBGtest_quotas (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:43201 method = POST path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31022025-04-30T06:38:17.736ZDEBGtest_quotas (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:43201 method = DELETE path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31032025-04-30T06:38:17.736ZDEBGtest_quotas (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:43201 method = GET path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31042025-04-30T06:38:17.736ZDEBGtest_quotas (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:43201 method = GET path = /crucible/0/regions/{id}/snapshots server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31052025-04-30T06:38:17.736ZDEBGtest_quotas (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:43201 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31062025-04-30T06:38:17.736ZDEBGtest_quotas (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:43201 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31072025-04-30T06:38:17.736ZDEBGtest_quotas (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:43201 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31082025-04-30T06:38:17.736ZDEBGtest_quotas (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:43201 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31092025-04-30T06:38:17.736ZINFOtest_quotas (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:43201 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31102025-04-30T06:38:17.737ZDEBGtest_quotas (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:43201 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31112025-04-30T06:38:17.737ZINFOtest_quotas (SledAgent): Created Simulated Crucible Server address = [::1]:43201 kind = storage server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31122025-04-30T06:38:17.737ZINFOtest_quotas (nexus): inserting test physical disk model = test-model physical_disk_id = 47d45d5d-4993-417b-af19-bb33cad16986 serial = totally-unique-serial: 47d45d5d-4993-417b-af19-bb33cad16986 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31132025-04-30T06:38:17.737ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31142025-04-30T06:38:17.737ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31152025-04-30T06:38:17.738ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31162025-04-30T06:38:17.738ZINFOtest_quotas (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
31172025-04-30T06:38:17.798ZINFOtest_quotas (dropshot_internal): accepted connection local_addr = 127.0.0.1:64801 remote_addr = 127.0.0.1:34994
31182025-04-30T06:38:17.798ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
31192025-04-30T06:38:17.799ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31202025-04-30T06:38:17.799ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 74d319ef-728f-4c05-a5f1-ac7b9ab04f4f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31212025-04-30T06:38:17.801ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:34994 req_id = 9157ff11-08fc-464d-b7a4-266436697481 uri = /metrics/producers
31222025-04-30T06:38:17.802ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 74d319ef-728f-4c05-a5f1-ac7b9ab04f4f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31232025-04-30T06:38:17.802ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 74d319ef-728f-4c05-a5f1-ac7b9ab04f4f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31242025-04-30T06:38:17.803ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 74d319ef-728f-4c05-a5f1-ac7b9ab04f4f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31252025-04-30T06:38:17.804ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:34994 req_id = 9157ff11-08fc-464d-b7a4-266436697481 resource = Database uri = /metrics/producers
31262025-04-30T06:38:17.804ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:34994 req_id = 9157ff11-08fc-464d-b7a4-266436697481 roles = RoleSet { roles: {} } uri = /metrics/producers
31272025-04-30T06:38:17.804ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:34994 req_id = 9157ff11-08fc-464d-b7a4-266436697481 resource = Database result = Ok(()) uri = /metrics/producers
31282025-04-30T06:38:17.805ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31292025-04-30T06:38:17.805ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31302025-04-30T06:38:17.805ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31312025-04-30T06:38:17.809ZINFOtest_quotas (dropshot_internal): request completed latency_us = 9155 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 74d319ef-728f-4c05-a5f1-ac7b9ab04f4f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31322025-04-30T06:38:17.809ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31332025-04-30T06:38:17.809ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31342025-04-30T06:38:17.809ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31352025-04-30T06:38:17.809ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31362025-04-30T06:38:17.810ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31372025-04-30T06:38:17.810ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "74d319ef-728f-4c05-a5f1-ac7b9ab04f4f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
31382025-04-30T06:38:17.810ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
31392025-04-30T06:38:17.815ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31402025-04-30T06:38:17.833ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31412025-04-30T06:38:17.866ZINFOtest_quotas (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
31422025-04-30T06:38:17.866ZINFOtest_quotas (dropshot_internal): request completed latency_us = 64868 local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:34994 req_id = 9157ff11-08fc-464d-b7a4-266436697481 response_code = 201 uri = /metrics/producers
31432025-04-30T06:38:17.967ZINFOtest_quotas (nexus): upserting zpool physical_disk_id = 47d45d5d-4993-417b-af19-bb33cad16986 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 885f41e2-3c1e-4eb6-8f80-76531de7b6f2
31442025-04-30T06:38:17.967ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31452025-04-30T06:38:17.967ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31462025-04-30T06:38:17.969ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31472025-04-30T06:38:17.973ZTRACtest_quotas (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31482025-04-30T06:38:17.973ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31492025-04-30T06:38:17.973ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31502025-04-30T06:38:17.973ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31512025-04-30T06:38:17.974ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31522025-04-30T06:38:17.987ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31532025-04-30T06:38:17.996ZDEBGtest_quotas (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31542025-04-30T06:38:18.013ZINFOtest_quotas (nexus): upserting Crucible dataset dataset_id = babd2712-e812-40ec-bf99-ce0bb3d718d7 zpool_id = 885f41e2-3c1e-4eb6-8f80-76531de7b6f2
31552025-04-30T06:38:18.044ZINFOtest_quotas (nexus): inserting test physical disk model = test-model physical_disk_id = 47d45d5d-4993-417b-af19-bb33cad16986 serial = totally-unique-serial: 47d45d5d-4993-417b-af19-bb33cad16986 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31562025-04-30T06:38:18.044ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31572025-04-30T06:38:18.044ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31582025-04-30T06:38:18.045ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31592025-04-30T06:38:18.051ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 47d45d5d-4993-417b-af19-bb33cad16986 (physical_disk), lookup_type: ById(47d45d5d-4993-417b-af19-bb33cad16986) }
31602025-04-30T06:38:18.051ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31612025-04-30T06:38:18.052ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31622025-04-30T06:38:18.052ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31632025-04-30T06:38:18.052ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31642025-04-30T06:38:18.056ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31652025-04-30T06:38:18.066ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 47d45d5d-4993-417b-af19-bb33cad16986 (physical_disk), lookup_type: ById(47d45d5d-4993-417b-af19-bb33cad16986) } result = Ok(())
31662025-04-30T06:38:18.066ZINFOtest_quotas (nexus): upserting zpool physical_disk_id = 47d45d5d-4993-417b-af19-bb33cad16986 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 885f41e2-3c1e-4eb6-8f80-76531de7b6f2
31672025-04-30T06:38:18.066ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31682025-04-30T06:38:18.066ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31692025-04-30T06:38:18.067ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31702025-04-30T06:38:18.068ZTRACtest_quotas (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31712025-04-30T06:38:18.068ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31722025-04-30T06:38:18.068ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31732025-04-30T06:38:18.068ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31742025-04-30T06:38:18.069ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31752025-04-30T06:38:18.074ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31762025-04-30T06:38:18.077ZDEBGtest_quotas (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31772025-04-30T06:38:18.085ZTRACtest_quotas (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
31782025-04-30T06:38:18.085ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31792025-04-30T06:38:18.085ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31802025-04-30T06:38:18.085ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31812025-04-30T06:38:18.086ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31822025-04-30T06:38:18.104ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31832025-04-30T06:38:18.110ZDEBGtest_quotas (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
31842025-04-30T06:38:18.124ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31852025-04-30T06:38:18.124ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31862025-04-30T06:38:18.125ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31872025-04-30T06:38:18.125ZTRACtest_quotas (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
31882025-04-30T06:38:18.125ZTRACtest_quotas (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
31892025-04-30T06:38:18.150ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
31902025-04-30T06:38:18.150ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31912025-04-30T06:38:18.150ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31922025-04-30T06:38:18.150ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
31932025-04-30T06:38:18.151ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31942025-04-30T06:38:18.151ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
31952025-04-30T06:38:18.151ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
31962025-04-30T06:38:18.151ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31972025-04-30T06:38:18.151ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
31982025-04-30T06:38:18.151ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
31992025-04-30T06:38:18.152ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32002025-04-30T06:38:18.154ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32012025-04-30T06:38:18.161ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
32022025-04-30T06:38:18.162ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32032025-04-30T06:38:18.162ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
32042025-04-30T06:38:18.162ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32052025-04-30T06:38:18.165ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32062025-04-30T06:38:18.170ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
32072025-04-30T06:38:18.170ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32082025-04-30T06:38:18.170ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32092025-04-30T06:38:18.171ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32102025-04-30T06:38:18.175ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32112025-04-30T06:38:18.175ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
32122025-04-30T06:38:18.176ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32132025-04-30T06:38:18.176ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
32142025-04-30T06:38:18.177ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
32152025-04-30T06:38:18.177ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b1 e6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62165 req_id = a6f0887f-8eb9-4c8f-ad8b-8f9ec2514c1f
32162025-04-30T06:38:18.177ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45542,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62165 req_id = a6f0887f-8eb9-4c8f-ad8b-8f9ec2514c1f
32172025-04-30T06:38:18.178ZDEBGtest_quotas (store): query key key = _mgs._tcp
32182025-04-30T06:38:18.178ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32192025-04-30T06:38:18.178ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62165 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a6f0887f-8eb9-4c8f-ad8b-8f9ec2514c1f
32202025-04-30T06:38:18.179ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3962, tv_nsec: 978277894 } })
32212025-04-30T06:38:18.179ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
32222025-04-30T06:38:18.180ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: cf a4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54352 req_id = d2e198c3-1bcb-4881-b33b-2953ebb4f572
32232025-04-30T06:38:18.180ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53156,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54352 req_id = d2e198c3-1bcb-4881-b33b-2953ebb4f572
32242025-04-30T06:38:18.180ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32252025-04-30T06:38:18.180ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54352 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d2e198c3-1bcb-4881-b33b-2953ebb4f572
32262025-04-30T06:38:18.250ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
32272025-04-30T06:38:18.250ZDEBGtest_quotas (producer-registration-task): will register with Nexus at 127.0.0.1:64801
32282025-04-30T06:38:18.317ZDEBGtest_quotas (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:64801/metrics/producers
32292025-04-30T06:38:18.319ZINFOtest_quotas (dropshot_internal): accepted connection local_addr = 127.0.0.1:64801 remote_addr = 127.0.0.1:51458
32302025-04-30T06:38:18.319ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: de 21 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .!..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52597 req_id = 2b0fd09e-09ad-4eb9-af1f-a0c1e3ceaa53
32312025-04-30T06:38:18.319ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56865,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52597 req_id = 2b0fd09e-09ad-4eb9-af1f-a0c1e3ceaa53
32322025-04-30T06:38:18.320ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
32332025-04-30T06:38:18.320ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56865, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52597 req_id = 2b0fd09e-09ad-4eb9-af1f-a0c1e3ceaa53
32342025-04-30T06:38:18.320ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
32352025-04-30T06:38:18.320ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
32362025-04-30T06:38:18.320ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
32372025-04-30T06:38:18.321ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:51458 req_id = d5b4f86b-9164-46f1-b61e-a66ef5e6618d uri = /metrics/producers
32382025-04-30T06:38:18.322ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:51458 req_id = d5b4f86b-9164-46f1-b61e-a66ef5e6618d resource = Database uri = /metrics/producers
32392025-04-30T06:38:18.322ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:51458 req_id = d5b4f86b-9164-46f1-b61e-a66ef5e6618d roles = RoleSet { roles: {} } uri = /metrics/producers
32402025-04-30T06:38:18.323ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:51458 req_id = d5b4f86b-9164-46f1-b61e-a66ef5e6618d resource = Database result = Ok(()) uri = /metrics/producers
32412025-04-30T06:38:18.365ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "655bf7e0-7eb3-44b0-aafc-11eb7ab8d0f8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32422025-04-30T06:38:18.365ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
32432025-04-30T06:38:18.376ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b3a6d49-c518-478e-a3a3-8b09bd548a3c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32442025-04-30T06:38:18.377ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
32452025-04-30T06:38:18.388ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5946f99b-b02e-4433-ae31-e00b80556a49", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32462025-04-30T06:38:18.388ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
32472025-04-30T06:38:18.390ZINFOtest_quotas (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
32482025-04-30T06:38:18.391ZINFOtest_quotas (dropshot_internal): request completed latency_us = 69692 local_addr = 127.0.0.1:64801 method = POST remote_addr = 127.0.0.1:51458 req_id = d5b4f86b-9164-46f1-b61e-a66ef5e6618d response_code = 201 uri = /metrics/producers
32492025-04-30T06:38:18.391ZDEBGtest_quotas (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "d5b4f86b-9164-46f1-b61e-a66ef5e6618d", "content-length": "41", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32502025-04-30T06:38:18.393ZDEBGtest_quotas (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
32512025-04-30T06:38:18.400ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ef9128b-d713-4e3e-9679-6cd18d2a7e24", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32522025-04-30T06:38:18.400ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
32532025-04-30T06:38:18.406ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2422960-d8a5-4d98-9c02-c2a9f6ddc9d8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32542025-04-30T06:38:18.411ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
32552025-04-30T06:38:18.415ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08b2a1bd-3a8c-4fc7-ade9-41619d5d619a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32562025-04-30T06:38:18.415ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
32572025-04-30T06:38:18.426ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "120c3efe-a7e0-4c27-a212-570ddfa70b24", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32582025-04-30T06:38:18.426ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
32592025-04-30T06:38:18.431ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6401c99b-a540-44f2-a2f7-b0b4ee9b1c6a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32602025-04-30T06:38:18.431ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
32612025-04-30T06:38:18.440ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
32622025-04-30T06:38:18.440ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eed8926-8714-4282-939b-9179b2579b4f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32632025-04-30T06:38:18.440ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
32642025-04-30T06:38:18.444ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cb371e7-6a69-4d9a-8073-5107b2b4d43c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32652025-04-30T06:38:18.444ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
32662025-04-30T06:38:18.449ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4b5d17f-cea9-4cbd-8362-3785a7f1f88e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32672025-04-30T06:38:18.449ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
32682025-04-30T06:38:18.451ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9387cb9e-6fe9-48dd-9d01-fd6bd77c5b9d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32692025-04-30T06:38:18.451ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
32702025-04-30T06:38:18.456ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d13395c3-0dc1-4c54-92ef-d37b1996eb74", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32712025-04-30T06:38:18.456ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
32722025-04-30T06:38:18.460ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "006dd4cb-d7e1-46e9-b077-2ea5d8041e4b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32732025-04-30T06:38:18.460ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
32742025-04-30T06:38:18.464ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b782de5a-1aca-40c5-950b-5c9de7732afa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32752025-04-30T06:38:18.464ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
32762025-04-30T06:38:18.467ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4941816e-bc0b-40a9-8f6e-0916aff689e8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32772025-04-30T06:38:18.467ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
32782025-04-30T06:38:18.470ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c19bd059-e2e9-440a-a13c-3ffc3b82f71d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32792025-04-30T06:38:18.470ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
32802025-04-30T06:38:18.473ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dec5eca-bec5-4afd-ad39-1a293e826896", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32812025-04-30T06:38:18.473ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
32822025-04-30T06:38:18.475ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39326949-b800-4f6d-afd0-fc8dde0d2f6e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32832025-04-30T06:38:18.476ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
32842025-04-30T06:38:18.477ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa9b7d1f-0994-4cd3-a95a-12ae91d33f47", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32852025-04-30T06:38:18.477ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
32862025-04-30T06:38:18.478ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85bbfed1-9a4e-450b-8f61-63a5a042f93d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32872025-04-30T06:38:18.478ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
32882025-04-30T06:38:18.479ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54388587-c835-4d7f-b82f-ced6f8a88fea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32892025-04-30T06:38:18.479ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
32902025-04-30T06:38:18.480ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "691e9923-470b-4792-82c9-8b15efcb239d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32912025-04-30T06:38:18.480ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
32922025-04-30T06:38:18.483ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d68583d1-fd01-4037-b28b-c28134f01907", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32932025-04-30T06:38:18.484ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
32942025-04-30T06:38:18.485ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27d6e89b-6de8-4448-80d1-45f6e97e216d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32952025-04-30T06:38:18.485ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
32962025-04-30T06:38:18.487ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e91dcf09-4f16-474f-94c0-a485d75d1b90", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32972025-04-30T06:38:18.488ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
32982025-04-30T06:38:18.490ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25387890-9fa1-4bdd-816e-8b73a95dc577", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
32992025-04-30T06:38:18.490ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
33002025-04-30T06:38:18.491ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33012025-04-30T06:38:18.491ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33022025-04-30T06:38:18.491ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33032025-04-30T06:38:18.491ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
33042025-04-30T06:38:18.492ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33052025-04-30T06:38:18.495ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfc25250-3224-452b-8754-20f5dd83a45d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33062025-04-30T06:38:18.495ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
33072025-04-30T06:38:18.495ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33082025-04-30T06:38:18.503ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33092025-04-30T06:38:18.505ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33102025-04-30T06:38:18.505ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
33112025-04-30T06:38:18.506ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33122025-04-30T06:38:18.523ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33132025-04-30T06:38:18.523ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
33142025-04-30T06:38:18.526ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33152025-04-30T06:38:18.533ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09722392-625a-414b-9528-1f43fd545023", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33162025-04-30T06:38:18.533ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
33172025-04-30T06:38:18.537ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56ba16df-6ce8-4637-b22d-aae003e8d56e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33182025-04-30T06:38:18.537ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
33192025-04-30T06:38:18.539ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79ea1adc-2f12-4b0c-a9c5-563d2e50d977", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33202025-04-30T06:38:18.539ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
33212025-04-30T06:38:18.543ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b250f1e-9b8d-4bbf-99ca-cf5ba73f0156", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33222025-04-30T06:38:18.543ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
33232025-04-30T06:38:18.545ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca205fd7-4120-418b-8168-45deefb434a2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33242025-04-30T06:38:18.545ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
33252025-04-30T06:38:18.546ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f89ddb9-ff07-4958-ae74-42f18c06081e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33262025-04-30T06:38:18.546ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
33272025-04-30T06:38:18.548ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14f93f12-7f3c-4f6b-840d-30c329b0dd59", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33282025-04-30T06:38:18.548ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
33292025-04-30T06:38:18.557ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f896fcbf-0926-41b0-9aa3-ed6d9bd828b3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33302025-04-30T06:38:18.557ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
33312025-04-30T06:38:18.559ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec5272b3-833b-44f3-97ec-f43f7e922bda", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33322025-04-30T06:38:18.559ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
33332025-04-30T06:38:18.564ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb080a73-0ebb-4d92-9590-a2bfea1bc5f8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33342025-04-30T06:38:18.564ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
33352025-04-30T06:38:18.565ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
33362025-04-30T06:38:18.568ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a156962-1eaf-4aa5-af4d-3ec2d50bf9ed", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33372025-04-30T06:38:18.569ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
33382025-04-30T06:38:18.571ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ec187a6-c87e-489f-81c0-fb6766859d13", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33392025-04-30T06:38:18.571ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
33402025-04-30T06:38:18.575ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d1d0df2-3c93-4ba2-ad0b-99f23e22fee7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33412025-04-30T06:38:18.575ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
33422025-04-30T06:38:18.576ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b711bfb2-b4f4-4f9d-9701-d5d458540945", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33432025-04-30T06:38:18.576ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
33442025-04-30T06:38:18.577ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5eee064d-dab1-4410-80ee-f4e48ee593dd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33452025-04-30T06:38:18.577ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
33462025-04-30T06:38:18.579ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f91188b9-8c41-412d-975b-e2c2c56e6cd7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33472025-04-30T06:38:18.579ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
33482025-04-30T06:38:18.581ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "501dfaf5-4f17-4f64-9a42-d8a820028748", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33492025-04-30T06:38:18.581ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33502025-04-30T06:38:18.581ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33512025-04-30T06:38:18.581ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33522025-04-30T06:38:18.582ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33532025-04-30T06:38:18.582ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33542025-04-30T06:38:18.587ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33552025-04-30T06:38:18.595ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33562025-04-30T06:38:18.595ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
33572025-04-30T06:38:18.595ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33582025-04-30T06:38:18.595ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33592025-04-30T06:38:18.595ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33602025-04-30T06:38:18.596ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33612025-04-30T06:38:18.598ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33622025-04-30T06:38:18.603ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
33632025-04-30T06:38:18.603ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33642025-04-30T06:38:18.603ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33652025-04-30T06:38:18.603ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33662025-04-30T06:38:18.662ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
33672025-04-30T06:38:18.662ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
33682025-04-30T06:38:18.663ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33692025-04-30T06:38:18.663ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33702025-04-30T06:38:18.663ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33712025-04-30T06:38:18.663ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
33722025-04-30T06:38:18.663ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33732025-04-30T06:38:18.663ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:62355 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
33742025-04-30T06:38:18.666ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:62355 req_id = 71fbef94-4914-4a60-8933-88ac54b030e7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33752025-04-30T06:38:18.667ZINFOtest_quotas (dropshot): request completed latency_us = 1239 local_addr = [::1]:55655 method = GET remote_addr = [::1]:62355 req_id = 71fbef94-4914-4a60-8933-88ac54b030e7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
33762025-04-30T06:38:18.668ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "71fbef94-4914-4a60-8933-88ac54b030e7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:17 GMT"} })
33772025-04-30T06:38:18.674ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33782025-04-30T06:38:18.674ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
33792025-04-30T06:38:18.674ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
33802025-04-30T06:38:18.679ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
33812025-04-30T06:38:18.679ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33822025-04-30T06:38:18.679ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33832025-04-30T06:38:18.679ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33842025-04-30T06:38:18.681ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33852025-04-30T06:38:18.682ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33862025-04-30T06:38:18.693ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33872025-04-30T06:38:18.694ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33882025-04-30T06:38:18.694ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
33892025-04-30T06:38:18.694ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33902025-04-30T06:38:18.697ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33912025-04-30T06:38:18.704ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
33922025-04-30T06:38:18.705ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
33932025-04-30T06:38:18.705ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
33942025-04-30T06:38:18.705ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
33952025-04-30T06:38:18.707ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33962025-04-30T06:38:18.707ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
33972025-04-30T06:38:18.708ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33982025-04-30T06:38:18.782ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
33992025-04-30T06:38:18.782ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
34002025-04-30T06:38:18.783ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
34012025-04-30T06:38:18.783ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 631.60403ms iteration = 4 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
34022025-04-30T06:38:18.783ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
34032025-04-30T06:38:18.783ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34042025-04-30T06:38:18.783ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34052025-04-30T06:38:18.783ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34062025-04-30T06:38:18.783ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34072025-04-30T06:38:18.784ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34082025-04-30T06:38:18.788ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34092025-04-30T06:38:18.792ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34102025-04-30T06:38:18.792ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34112025-04-30T06:38:18.792ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34122025-04-30T06:38:18.792ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34132025-04-30T06:38:18.828ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
34142025-04-30T06:38:18.828ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
34152025-04-30T06:38:18.829ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3d 8f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 =............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53019 req_id = 7fd584ae-5e2c-45d8-8ced-7e3bd94119b1
34162025-04-30T06:38:18.829ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15759,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53019 req_id = 7fd584ae-5e2c-45d8-8ced-7e3bd94119b1
34172025-04-30T06:38:18.830ZDEBGtest_quotas (store): query key key = _mgs._tcp
34182025-04-30T06:38:18.830ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34192025-04-30T06:38:18.830ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53019 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7fd584ae-5e2c-45d8-8ced-7e3bd94119b1
34202025-04-30T06:38:18.830ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3963, tv_nsec: 629684930 } })
34212025-04-30T06:38:18.830ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
34222025-04-30T06:38:18.831ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 56 d9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e V...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60542 req_id = 3bc8298c-edb8-4114-81b4-e599f6b8404e
34232025-04-30T06:38:18.842ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22233,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60542 req_id = 3bc8298c-edb8-4114-81b4-e599f6b8404e
34242025-04-30T06:38:18.842ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34252025-04-30T06:38:18.842ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60542 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3bc8298c-edb8-4114-81b4-e599f6b8404e
34262025-04-30T06:38:18.847ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34272025-04-30T06:38:18.848ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34282025-04-30T06:38:18.848ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34292025-04-30T06:38:18.848ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
34302025-04-30T06:38:18.849ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34312025-04-30T06:38:18.929ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
34322025-04-30T06:38:18.931ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6a ae 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c j............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55323 req_id = 439517c6-122a-4c03-b0e4-de0fc7cd3731
34332025-04-30T06:38:18.932ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27310,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55323 req_id = 439517c6-122a-4c03-b0e4-de0fc7cd3731
34342025-04-30T06:38:18.932ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
34352025-04-30T06:38:18.932ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27310, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55323 req_id = 439517c6-122a-4c03-b0e4-de0fc7cd3731
34362025-04-30T06:38:18.934ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
34372025-04-30T06:38:18.934ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
34382025-04-30T06:38:18.934ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
34392025-04-30T06:38:18.935ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34402025-04-30T06:38:18.946ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34412025-04-30T06:38:18.946ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34422025-04-30T06:38:18.946ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
34432025-04-30T06:38:18.947ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34442025-04-30T06:38:19.011ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34452025-04-30T06:38:19.011ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
34462025-04-30T06:38:19.012ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34472025-04-30T06:38:19.030ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d6f9782-c78f-401b-9040-79e24783ae79", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34482025-04-30T06:38:19.031ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
34492025-04-30T06:38:19.046ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6490414d-760a-41cc-a937-9df36b5dac33", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34502025-04-30T06:38:19.047ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
34512025-04-30T06:38:19.087ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9adfcb21-0e60-4ca5-8d9c-64147d2c9305", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34522025-04-30T06:38:19.088ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
34532025-04-30T06:38:19.092ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71c2876f-60b3-4da4-8f82-5500c6b843b8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34542025-04-30T06:38:19.094ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
34552025-04-30T06:38:19.098ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b8dac35-9f96-433c-9125-0702cb22c1ed", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34562025-04-30T06:38:19.099ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
34572025-04-30T06:38:19.105ZTRACtest_quotas (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
34582025-04-30T06:38:19.106ZTRACtest_quotas (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
34592025-04-30T06:38:19.107ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
34602025-04-30T06:38:19.109ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91fc8b4f-9dfa-44ed-aeea-ea47db32cd73", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34612025-04-30T06:38:19.109ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
34622025-04-30T06:38:19.111ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c167ff-65fc-4e39-954c-571da629fd26", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34632025-04-30T06:38:19.112ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
34642025-04-30T06:38:19.119ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "365fa967-08ea-4bba-8686-2ae30747998e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34652025-04-30T06:38:19.119ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
34662025-04-30T06:38:19.121ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba463ae1-a725-46ee-b0ea-768e2124055e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34672025-04-30T06:38:19.121ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
34682025-04-30T06:38:19.122ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f950f25c-2e71-4f0e-adb1-cd47209aefac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34692025-04-30T06:38:19.122ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
34702025-04-30T06:38:19.123ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6084029-18ce-4878-8b55-da585b8fad4d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34712025-04-30T06:38:19.124ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
34722025-04-30T06:38:19.128ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcc1cb94-6f3d-474d-a2c7-feea652e4e4b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34732025-04-30T06:38:19.128ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
34742025-04-30T06:38:19.130ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d08eac06-99de-4aa2-a4bf-9c6eced995f1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34752025-04-30T06:38:19.130ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
34762025-04-30T06:38:19.132ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "378f20c4-f202-43f8-a941-b496d96d8a01", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34772025-04-30T06:38:19.132ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
34782025-04-30T06:38:19.134ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2d00bef-7fa3-4e36-b098-82adfbfa1c60", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34792025-04-30T06:38:19.134ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
34802025-04-30T06:38:19.136ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4436ec3f-e8ad-45ba-83a4-11974cc5756c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34812025-04-30T06:38:19.136ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
34822025-04-30T06:38:19.137ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d8c2636-4bf8-4399-ab0a-af48483bc2e0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34832025-04-30T06:38:19.137ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
34842025-04-30T06:38:19.139ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da8bd6b1-f8f3-4a4a-b585-196ef5509657", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34852025-04-30T06:38:19.139ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
34862025-04-30T06:38:19.141ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "feeba912-99e6-4233-9876-dde67ccd968a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34872025-04-30T06:38:19.142ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
34882025-04-30T06:38:19.143ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bb33ad3-d0b5-41a8-89db-bece8723fec7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34892025-04-30T06:38:19.143ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
34902025-04-30T06:38:19.144ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ee4dbac-ad8c-4807-b64e-ecca623a958e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34912025-04-30T06:38:19.144ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
34922025-04-30T06:38:19.145ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e25ebcc-5ad6-4de0-9a95-e786faa9700f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34932025-04-30T06:38:19.145ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
34942025-04-30T06:38:19.146ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9d295e1-80c2-4915-9d28-e04ff899953e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34952025-04-30T06:38:19.146ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
34962025-04-30T06:38:19.147ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "482060c3-9373-4785-bfbb-43b5deecd788", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34972025-04-30T06:38:19.148ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
34982025-04-30T06:38:19.149ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dc3b0e9-0a08-4beb-b4c8-36a5fd0e48fd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
34992025-04-30T06:38:19.150ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
35002025-04-30T06:38:19.153ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "779bec3f-bbfb-4e35-9109-9966a6d64d3e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35012025-04-30T06:38:19.153ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
35022025-04-30T06:38:19.155ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8bc1707-aa94-4b1f-8074-d92dd584e309", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35032025-04-30T06:38:19.156ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
35042025-04-30T06:38:19.171ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35052025-04-30T06:38:19.172ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35062025-04-30T06:38:19.172ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35072025-04-30T06:38:19.172ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
35082025-04-30T06:38:19.173ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35092025-04-30T06:38:19.173ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3c3c18d-f5b4-4f3e-bb6a-80535cb56f19", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35102025-04-30T06:38:19.173ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
35112025-04-30T06:38:19.175ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7644232-e6ae-4b1d-99dd-ba5c07d608dc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35122025-04-30T06:38:19.176ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
35132025-04-30T06:38:19.176ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35142025-04-30T06:38:19.184ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35152025-04-30T06:38:19.184ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35162025-04-30T06:38:19.184ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
35172025-04-30T06:38:19.184ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35182025-04-30T06:38:19.188ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35192025-04-30T06:38:19.188ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
35202025-04-30T06:38:19.195ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35212025-04-30T06:38:19.195ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81c80108-957d-478d-88ac-9ded80f3dbe3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35222025-04-30T06:38:19.195ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
35232025-04-30T06:38:19.195ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d32d768a-2b68-4e4d-a4ac-e0692bfce2d9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35242025-04-30T06:38:19.195ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
35252025-04-30T06:38:19.196ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e48e7e36-31d9-4231-80b6-86d3a4800cf0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35262025-04-30T06:38:19.197ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
35272025-04-30T06:38:19.199ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "baafa9c7-afb6-43aa-b410-0bf45b27f993", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35282025-04-30T06:38:19.199ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
35292025-04-30T06:38:19.200ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a6db303-f446-42a1-9b13-53e50cae7375", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35302025-04-30T06:38:19.201ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
35312025-04-30T06:38:19.203ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5130183-be7e-48cd-b093-9ad3986eab69", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35322025-04-30T06:38:19.203ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
35332025-04-30T06:38:19.208ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48a88d55-d394-48c4-a4ae-32f28769204c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35342025-04-30T06:38:19.208ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
35352025-04-30T06:38:19.228ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1347b4f1-a321-4c4d-983c-876ab283b24a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35362025-04-30T06:38:19.228ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
35372025-04-30T06:38:19.240ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
35382025-04-30T06:38:19.241ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7af40f38-52d5-48d8-94aa-f56109dc1ad8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35392025-04-30T06:38:19.242ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
35402025-04-30T06:38:19.245ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b770d4fe-1c18-47f1-a29f-95ee1e095f1b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35412025-04-30T06:38:19.245ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
35422025-04-30T06:38:19.249ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98af7b76-4cc5-4438-b5ff-af8a1478be23", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35432025-04-30T06:38:19.249ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
35442025-04-30T06:38:19.251ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ae2dc83-c899-4022-8764-e76ef6ca5331", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35452025-04-30T06:38:19.252ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
35462025-04-30T06:38:19.252ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91802dcd-8c65-474e-b849-1b9b41b13709", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35472025-04-30T06:38:19.252ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
35482025-04-30T06:38:19.253ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0bf8b92-2605-4548-9ae0-afbfa3cf3070", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35492025-04-30T06:38:19.254ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
35502025-04-30T06:38:19.255ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d79823bb-ccfd-4056-b80c-a644cd5d39c6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35512025-04-30T06:38:19.255ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
35522025-04-30T06:38:19.256ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5d041df-2aea-46ff-b16d-c8322785453e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35532025-04-30T06:38:19.256ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35542025-04-30T06:38:19.256ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35552025-04-30T06:38:19.256ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35562025-04-30T06:38:19.256ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35572025-04-30T06:38:19.257ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35582025-04-30T06:38:19.270ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35592025-04-30T06:38:19.275ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35602025-04-30T06:38:19.276ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35612025-04-30T06:38:19.276ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35622025-04-30T06:38:19.276ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35632025-04-30T06:38:19.276ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35642025-04-30T06:38:19.276ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35652025-04-30T06:38:19.279ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35662025-04-30T06:38:19.284ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35672025-04-30T06:38:19.284ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35682025-04-30T06:38:19.284ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35692025-04-30T06:38:19.284ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35702025-04-30T06:38:19.297ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35712025-04-30T06:38:19.297ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35722025-04-30T06:38:19.297ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35732025-04-30T06:38:19.297ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
35742025-04-30T06:38:19.298ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35752025-04-30T06:38:19.371ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
35762025-04-30T06:38:19.372ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
35772025-04-30T06:38:19.372ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:41517 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
35782025-04-30T06:38:19.374ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:41517 req_id = f49ec798-283c-4e9e-b1e1-7a34aa4ee1e1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35792025-04-30T06:38:19.374ZINFOtest_quotas (dropshot): request completed latency_us = 547 local_addr = [::1]:55655 method = GET remote_addr = [::1]:41517 req_id = f49ec798-283c-4e9e-b1e1-7a34aa4ee1e1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35802025-04-30T06:38:19.374ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35812025-04-30T06:38:19.394ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35822025-04-30T06:38:19.394ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35832025-04-30T06:38:19.394ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
35842025-04-30T06:38:19.396ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35852025-04-30T06:38:19.396ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f49ec798-283c-4e9e-b1e1-7a34aa4ee1e1", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
35862025-04-30T06:38:19.398ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
35872025-04-30T06:38:19.398ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
35882025-04-30T06:38:19.398ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
35892025-04-30T06:38:19.401ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
35902025-04-30T06:38:19.401ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35912025-04-30T06:38:19.401ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35922025-04-30T06:38:19.401ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35932025-04-30T06:38:19.402ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35942025-04-30T06:38:19.408ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35952025-04-30T06:38:19.409ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
35962025-04-30T06:38:19.409ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35972025-04-30T06:38:19.410ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35982025-04-30T06:38:19.433ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
35992025-04-30T06:38:19.433ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36002025-04-30T06:38:19.433ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36012025-04-30T06:38:19.434ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36022025-04-30T06:38:19.525ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
36032025-04-30T06:38:19.564ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
36042025-04-30T06:38:19.564ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
36052025-04-30T06:38:19.564ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 781.431501ms iteration = 5 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
36062025-04-30T06:38:19.564ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
36072025-04-30T06:38:19.564ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36082025-04-30T06:38:19.564ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36092025-04-30T06:38:19.564ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36102025-04-30T06:38:19.564ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36112025-04-30T06:38:19.566ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36122025-04-30T06:38:19.575ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36132025-04-30T06:38:19.579ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36142025-04-30T06:38:19.579ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36152025-04-30T06:38:19.579ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36162025-04-30T06:38:19.582ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36172025-04-30T06:38:19.583ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36182025-04-30T06:38:19.583ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36192025-04-30T06:38:19.583ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36202025-04-30T06:38:19.583ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
36212025-04-30T06:38:19.585ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36222025-04-30T06:38:19.588ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
36232025-04-30T06:38:19.589ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
36242025-04-30T06:38:19.589ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36252025-04-30T06:38:19.599ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36262025-04-30T06:38:19.599ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36272025-04-30T06:38:19.599ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
36282025-04-30T06:38:19.600ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36292025-04-30T06:38:19.603ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e5 f0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50630 req_id = 6f3410a5-c039-47ab-bfbd-534664f5a804
36302025-04-30T06:38:19.603ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58864,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50630 req_id = 6f3410a5-c039-47ab-bfbd-534664f5a804
36312025-04-30T06:38:19.604ZDEBGtest_quotas (store): query key key = _mgs._tcp
36322025-04-30T06:38:19.604ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36332025-04-30T06:38:19.604ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50630 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6f3410a5-c039-47ab-bfbd-534664f5a804
36342025-04-30T06:38:19.604ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36352025-04-30T06:38:19.604ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
36362025-04-30T06:38:19.605ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36372025-04-30T06:38:19.606ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3964, tv_nsec: 405170224 } })
36382025-04-30T06:38:19.606ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
36392025-04-30T06:38:19.610ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: cb 7b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .{..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65381 req_id = fcae8c78-7af7-4a58-9ae9-c5aa22af5fe2
36402025-04-30T06:38:19.610ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52091,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65381 req_id = fcae8c78-7af7-4a58-9ae9-c5aa22af5fe2
36412025-04-30T06:38:19.610ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36422025-04-30T06:38:19.610ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65381 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fcae8c78-7af7-4a58-9ae9-c5aa22af5fe2
36432025-04-30T06:38:19.681ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
36442025-04-30T06:38:19.684ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9a cd 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56630 req_id = 9eb16edd-ecc0-484c-8a0e-831e3eedcb73
36452025-04-30T06:38:19.684ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39629,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56630 req_id = 9eb16edd-ecc0-484c-8a0e-831e3eedcb73
36462025-04-30T06:38:19.685ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
36472025-04-30T06:38:19.685ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39629, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56630 req_id = 9eb16edd-ecc0-484c-8a0e-831e3eedcb73
36482025-04-30T06:38:19.685ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
36492025-04-30T06:38:19.685ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
36502025-04-30T06:38:19.685ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
36512025-04-30T06:38:19.687ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cbce972-742f-4225-a594-bac360dd970b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36522025-04-30T06:38:19.688ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
36532025-04-30T06:38:19.689ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ad419e7-b4ba-4a2c-88b7-36d263b12217", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36542025-04-30T06:38:19.690ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
36552025-04-30T06:38:19.695ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d453f7ce-a6bf-41f7-9bc5-17d22d034054", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36562025-04-30T06:38:19.695ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
36572025-04-30T06:38:19.697ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4296dc6-dbf3-4e69-bd74-30174b227ee5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36582025-04-30T06:38:19.697ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
36592025-04-30T06:38:19.702ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4f2eba0-4811-47f8-8f3c-736d18c8ee11", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36602025-04-30T06:38:19.702ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
36612025-04-30T06:38:19.705ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7e46dc7-45ab-4db7-b96f-a7cec3e7e2d2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36622025-04-30T06:38:19.705ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
36632025-04-30T06:38:19.707ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d31cab52-006a-421a-ae05-8d82cb3a50e7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36642025-04-30T06:38:19.707ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
36652025-04-30T06:38:19.712ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0ccbf88-749c-48a9-a9b9-c0a130376dba", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36662025-04-30T06:38:19.712ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
36672025-04-30T06:38:19.716ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fd53581-93dc-404f-9310-9c552ca1303e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36682025-04-30T06:38:19.716ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
36692025-04-30T06:38:19.718ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "176d2656-fd3d-47e5-be32-00eb5f58dae7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36702025-04-30T06:38:19.718ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
36712025-04-30T06:38:19.719ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58c63a2c-e2c1-4e78-8e36-8bc07192f265", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36722025-04-30T06:38:19.719ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
36732025-04-30T06:38:19.720ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17feeca5-c160-46a9-980b-85439e5a0f07", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36742025-04-30T06:38:19.720ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
36752025-04-30T06:38:19.722ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "541a943f-8fbf-48c8-b675-ba98bb313e1c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36762025-04-30T06:38:19.722ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
36772025-04-30T06:38:19.725ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
36782025-04-30T06:38:19.735ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9424133c-d8e2-43b4-8d2f-87461f656b97", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36792025-04-30T06:38:19.736ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
36802025-04-30T06:38:19.739ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
36812025-04-30T06:38:19.739ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36822025-04-30T06:38:19.740ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 8605f60a-09d0-4cb0-b297-4224a639637b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36832025-04-30T06:38:19.740ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 8605f60a-09d0-4cb0-b297-4224a639637b resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36842025-04-30T06:38:19.740ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 8605f60a-09d0-4cb0-b297-4224a639637b roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36852025-04-30T06:38:19.746ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 8605f60a-09d0-4cb0-b297-4224a639637b resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36862025-04-30T06:38:19.748ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12a8dd50-d24d-467f-a0c9-05a632dc3bc8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36872025-04-30T06:38:19.749ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
36882025-04-30T06:38:19.752ZINFOtest_quotas (dropshot_internal): request completed latency_us = 11868 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 8605f60a-09d0-4cb0-b297-4224a639637b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36892025-04-30T06:38:19.752ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "8605f60a-09d0-4cb0-b297-4224a639637b", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36902025-04-30T06:38:19.753ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
36912025-04-30T06:38:19.753ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 18d583b9-7316-4491-b9e8-d96dda804cca uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
36922025-04-30T06:38:19.754ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 18d583b9-7316-4491-b9e8-d96dda804cca resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
36932025-04-30T06:38:19.754ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 18d583b9-7316-4491-b9e8-d96dda804cca roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
36942025-04-30T06:38:19.767ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 18d583b9-7316-4491-b9e8-d96dda804cca resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
36952025-04-30T06:38:19.771ZINFOtest_quotas (dropshot_internal): request completed latency_us = 17794 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 18d583b9-7316-4491-b9e8-d96dda804cca response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
36962025-04-30T06:38:19.781ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "18d583b9-7316-4491-b9e8-d96dda804cca", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
36972025-04-30T06:38:19.781ZDEBGtest_quotas (oximeter-agent): registered new metric producer address = 127.0.0.1:48722 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
36982025-04-30T06:38:19.839ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
36992025-04-30T06:38:19.840ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37002025-04-30T06:38:19.840ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37012025-04-30T06:38:19.840ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37022025-04-30T06:38:19.840ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
37032025-04-30T06:38:19.848ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37042025-04-30T06:38:19.849ZDEBGtest_quotas (oximeter-agent): registered new metric producer address = [::1]:42959 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
37052025-04-30T06:38:19.849ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7569d3a2-030b-4078-b882-4c67d6518c12", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37062025-04-30T06:38:19.849ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
37072025-04-30T06:38:19.888ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
37082025-04-30T06:38:19.890ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
37092025-04-30T06:38:19.890ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37102025-04-30T06:38:19.896ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37112025-04-30T06:38:19.896ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37122025-04-30T06:38:19.896ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
37132025-04-30T06:38:19.896ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37142025-04-30T06:38:19.898ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee6feb5d-e30d-4bdf-ba96-1ceda14418fb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37152025-04-30T06:38:19.898ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
37162025-04-30T06:38:19.900ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37172025-04-30T06:38:19.900ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
37182025-04-30T06:38:19.901ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37192025-04-30T06:38:19.901ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79bb5f1f-cfd3-4fb0-8356-03947860b2fb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37202025-04-30T06:38:19.901ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
37212025-04-30T06:38:19.904ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e80afae-afac-497c-9381-0ce6cec5a14a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37222025-04-30T06:38:19.904ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
37232025-04-30T06:38:19.906ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dd07e24-5fe8-458a-a81f-0f0ce100ec87", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37242025-04-30T06:38:19.906ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
37252025-04-30T06:38:19.907ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92e5c899-ae6f-4c40-8f37-44851f21a37c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37262025-04-30T06:38:19.908ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
37272025-04-30T06:38:19.909ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c39a2f28-58ed-435c-8e6f-0e734b7c5958", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37282025-04-30T06:38:19.909ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
37292025-04-30T06:38:19.912ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2080e722-fd47-478e-b07d-4887454615dc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37302025-04-30T06:38:19.912ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
37312025-04-30T06:38:19.914ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16869dc0-9b30-477e-95a5-83eba3c38503", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37322025-04-30T06:38:19.914ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
37332025-04-30T06:38:19.916ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8f6387c-7798-4027-b2de-df3dee23851f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37342025-04-30T06:38:19.917ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
37352025-04-30T06:38:19.919ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fad6384a-c27d-46fe-b07d-f2104a77cdc6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37362025-04-30T06:38:19.919ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
37372025-04-30T06:38:19.930ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eec6cd08-ec93-460a-9efb-c9be4e51a215", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:18 GMT"} })
37382025-04-30T06:38:19.930ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
37392025-04-30T06:38:19.935ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
37402025-04-30T06:38:19.953ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6979e83a-245d-4e1e-8d66-c2cfcd0f66a6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37412025-04-30T06:38:19.953ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
37422025-04-30T06:38:19.957ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "991da018-e773-4de5-84d8-abb84cd78994", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37432025-04-30T06:38:19.957ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
37442025-04-30T06:38:19.959ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e83c4c27-3ded-4514-aed8-f08b6d73cae9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37452025-04-30T06:38:19.959ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
37462025-04-30T06:38:19.960ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d80944a-73d8-4799-95ba-bd70193c22e7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37472025-04-30T06:38:19.960ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
37482025-04-30T06:38:19.961ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f1ae763-1168-4c6a-81ae-6f80d1aec6d8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37492025-04-30T06:38:19.961ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
37502025-04-30T06:38:19.963ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3435a20-8553-45c8-bd71-e50ef71381d7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37512025-04-30T06:38:19.963ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
37522025-04-30T06:38:19.964ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "286d6174-7282-40fe-bf35-65869cdfa6c1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37532025-04-30T06:38:19.964ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
37542025-04-30T06:38:19.965ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb623b50-4ca4-4d97-a8f9-81a549d41c62", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37552025-04-30T06:38:19.966ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
37562025-04-30T06:38:19.968ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d6255ab-9489-4961-aef4-3935392599b6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37572025-04-30T06:38:19.968ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
37582025-04-30T06:38:19.969ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d22bb8cf-2000-4d4a-a88f-212104008b09", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37592025-04-30T06:38:19.969ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
37602025-04-30T06:38:19.971ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "febe55b5-acac-431c-aac0-13bcb681d0bb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37612025-04-30T06:38:19.972ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
37622025-04-30T06:38:19.973ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea3efc5d-72b7-4ebf-a88b-a4e918add1be", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37632025-04-30T06:38:19.973ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
37642025-04-30T06:38:19.975ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "832ddae3-3e28-486f-8238-16275c2207f4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37652025-04-30T06:38:19.975ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
37662025-04-30T06:38:19.977ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cecc8661-df31-4b47-ab30-acbfcca30cea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37672025-04-30T06:38:19.977ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
37682025-04-30T06:38:19.978ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9d582d9-83db-4f73-afd4-df32a3b4b0f5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37692025-04-30T06:38:19.978ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
37702025-04-30T06:38:19.979ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16b8e31b-4589-46f9-a5c6-fd22301fae9e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37712025-04-30T06:38:19.979ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
37722025-04-30T06:38:19.980ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ad47934-6bb1-434d-a64b-a41a573d1daa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37732025-04-30T06:38:19.980ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
37742025-04-30T06:38:19.981ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d88b981-c85f-43af-974d-bfe320f799fd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
37752025-04-30T06:38:19.981ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37762025-04-30T06:38:19.981ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37772025-04-30T06:38:19.981ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37782025-04-30T06:38:19.981ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37792025-04-30T06:38:19.982ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37802025-04-30T06:38:19.985ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37812025-04-30T06:38:19.989ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37822025-04-30T06:38:19.989ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37832025-04-30T06:38:19.989ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37842025-04-30T06:38:19.989ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37852025-04-30T06:38:19.989ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37862025-04-30T06:38:19.990ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37872025-04-30T06:38:19.990ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37882025-04-30T06:38:19.990ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37892025-04-30T06:38:19.990ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37902025-04-30T06:38:19.990ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
37912025-04-30T06:38:19.991ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37922025-04-30T06:38:19.992ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37932025-04-30T06:38:20.004ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
37942025-04-30T06:38:20.004ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37952025-04-30T06:38:20.005ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37962025-04-30T06:38:20.006ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37972025-04-30T06:38:20.007ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37982025-04-30T06:38:20.013ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37992025-04-30T06:38:20.013ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38002025-04-30T06:38:20.013ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
38012025-04-30T06:38:20.014ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38022025-04-30T06:38:20.057ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
38032025-04-30T06:38:20.057ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
38042025-04-30T06:38:20.058ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:53771 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
38052025-04-30T06:38:20.059ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:53771 req_id = c8cf5874-9f94-45ba-8327-44cfd50ca9d3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38062025-04-30T06:38:20.059ZINFOtest_quotas (dropshot): request completed latency_us = 484 local_addr = [::1]:55655 method = GET remote_addr = [::1]:53771 req_id = c8cf5874-9f94-45ba-8327-44cfd50ca9d3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38072025-04-30T06:38:20.059ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38082025-04-30T06:38:20.059ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
38092025-04-30T06:38:20.060ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38102025-04-30T06:38:20.060ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8cf5874-9f94-45ba-8327-44cfd50ca9d3", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38112025-04-30T06:38:20.062ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38122025-04-30T06:38:20.062ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38132025-04-30T06:38:20.062ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
38142025-04-30T06:38:20.064ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38152025-04-30T06:38:20.065ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38162025-04-30T06:38:20.065ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38172025-04-30T06:38:20.065ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38182025-04-30T06:38:20.065ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38192025-04-30T06:38:20.068ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38202025-04-30T06:38:20.074ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38212025-04-30T06:38:20.075ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38222025-04-30T06:38:20.075ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38232025-04-30T06:38:20.076ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38242025-04-30T06:38:20.100ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
38252025-04-30T06:38:20.102ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
38262025-04-30T06:38:20.102ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
38272025-04-30T06:38:20.102ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 537.83461ms iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
38282025-04-30T06:38:20.102ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
38292025-04-30T06:38:20.102ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38302025-04-30T06:38:20.102ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38312025-04-30T06:38:20.102ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38322025-04-30T06:38:20.102ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38332025-04-30T06:38:20.103ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38342025-04-30T06:38:20.103ZTRACtest_quotas (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38352025-04-30T06:38:20.103ZTRACtest_quotas (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38362025-04-30T06:38:20.106ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38372025-04-30T06:38:20.111ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38382025-04-30T06:38:20.111ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38392025-04-30T06:38:20.111ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38402025-04-30T06:38:20.111ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38412025-04-30T06:38:20.114ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
38422025-04-30T06:38:20.115ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
38432025-04-30T06:38:20.115ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ea b3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58593 req_id = 358f1277-d03c-462a-abbe-5ba996fdff47
38442025-04-30T06:38:20.115ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60083,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58593 req_id = 358f1277-d03c-462a-abbe-5ba996fdff47
38452025-04-30T06:38:20.116ZDEBGtest_quotas (store): query key key = _mgs._tcp
38462025-04-30T06:38:20.116ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38472025-04-30T06:38:20.116ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58593 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 358f1277-d03c-462a-abbe-5ba996fdff47
38482025-04-30T06:38:20.116ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3964, tv_nsec: 915549991 } })
38492025-04-30T06:38:20.116ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
38502025-04-30T06:38:20.116ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 09 45 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .E..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55869 req_id = dc4766b2-9a58-4387-a6d4-aa5573842d3e
38512025-04-30T06:38:20.117ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2373,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55869 req_id = dc4766b2-9a58-4387-a6d4-aa5573842d3e
38522025-04-30T06:38:20.117ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38532025-04-30T06:38:20.117ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55869 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dc4766b2-9a58-4387-a6d4-aa5573842d3e
38542025-04-30T06:38:20.159ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
38552025-04-30T06:38:20.160ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38562025-04-30T06:38:20.160ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38572025-04-30T06:38:20.160ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38582025-04-30T06:38:20.160ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
38592025-04-30T06:38:20.160ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38602025-04-30T06:38:20.161ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c2 95 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50227 req_id = ed8c34cf-697d-439a-9e84-586776b6b4c9
38612025-04-30T06:38:20.161ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49813,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50227 req_id = ed8c34cf-697d-439a-9e84-586776b6b4c9
38622025-04-30T06:38:20.162ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
38632025-04-30T06:38:20.162ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49813, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50227 req_id = ed8c34cf-697d-439a-9e84-586776b6b4c9
38642025-04-30T06:38:20.162ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
38652025-04-30T06:38:20.162ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
38662025-04-30T06:38:20.162ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
38672025-04-30T06:38:20.163ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38682025-04-30T06:38:20.171ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38692025-04-30T06:38:20.171ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38702025-04-30T06:38:20.171ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
38712025-04-30T06:38:20.171ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38722025-04-30T06:38:20.173ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ee9a828-47e0-41a5-87b2-846c583e48bb", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38732025-04-30T06:38:20.173ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
38742025-04-30T06:38:20.174ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38752025-04-30T06:38:20.174ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
38762025-04-30T06:38:20.174ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38772025-04-30T06:38:20.176ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8d0265e-91a3-48fb-932c-0a0d9827feb5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38782025-04-30T06:38:20.176ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
38792025-04-30T06:38:20.178ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "681eb6e9-7d4e-4206-9766-1a3bb625fe38", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38802025-04-30T06:38:20.179ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
38812025-04-30T06:38:20.180ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab95f829-66b0-4623-b424-c3091b4070ed", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38822025-04-30T06:38:20.180ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
38832025-04-30T06:38:20.182ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87031867-3516-4ca3-ba2e-204469636390", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38842025-04-30T06:38:20.182ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
38852025-04-30T06:38:20.184ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ad781dd-f99a-4f09-813b-f229bd6d3362", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38862025-04-30T06:38:20.184ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
38872025-04-30T06:38:20.186ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c7c3607-c0d5-46ad-8997-448cb503b561", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38882025-04-30T06:38:20.186ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
38892025-04-30T06:38:20.188ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4429fba3-770b-4124-ae80-3d9b3370253f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38902025-04-30T06:38:20.188ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
38912025-04-30T06:38:20.189ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52da0127-cbd5-4861-b545-138a42eeeb5d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38922025-04-30T06:38:20.189ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
38932025-04-30T06:38:20.190ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e863157-12dd-4a38-8716-8f86c5c0e675", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38942025-04-30T06:38:20.191ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
38952025-04-30T06:38:20.192ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea636294-40fa-4cfe-9799-11e7479c2550", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38962025-04-30T06:38:20.192ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
38972025-04-30T06:38:20.201ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d945c2b-294f-4f85-a977-c8aa637bb129", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
38982025-04-30T06:38:20.201ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
38992025-04-30T06:38:20.204ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86dd5766-8842-4806-85da-7902ee74a03c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39002025-04-30T06:38:20.204ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
39012025-04-30T06:38:20.214ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
39022025-04-30T06:38:20.215ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af9683f3-4fa5-437c-a33b-bf0cfee44bce", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39032025-04-30T06:38:20.215ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
39042025-04-30T06:38:20.217ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5435d6ec-4ba9-448b-8672-2e23268a5957", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39052025-04-30T06:38:20.217ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
39062025-04-30T06:38:20.219ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea402f1c-d281-4fb1-b3ec-19f0f42f1a49", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39072025-04-30T06:38:20.219ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
39082025-04-30T06:38:20.221ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "955490d4-822b-474a-9a48-061130c0eb69", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39092025-04-30T06:38:20.221ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
39102025-04-30T06:38:20.223ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bac15f1-d2e0-413b-a5a5-1a90045d09ed", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39112025-04-30T06:38:20.223ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
39122025-04-30T06:38:20.225ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fdfff7d-b848-4d40-9627-59dc052adab6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39132025-04-30T06:38:20.225ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
39142025-04-30T06:38:20.226ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6417a415-1d93-401f-ab7d-b72a524722e8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39152025-04-30T06:38:20.227ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
39162025-04-30T06:38:20.228ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b405693d-9fb1-45c6-b889-cec4f7f5841a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39172025-04-30T06:38:20.228ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
39182025-04-30T06:38:20.229ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97efdd56-47e9-481c-9460-7680d195df19", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39192025-04-30T06:38:20.230ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
39202025-04-30T06:38:20.232ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da9f0817-7d71-4dad-beef-d55407c775d4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39212025-04-30T06:38:20.232ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
39222025-04-30T06:38:20.235ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "108dde78-e2d4-4078-b35e-28ca158f4fda", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39232025-04-30T06:38:20.236ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
39242025-04-30T06:38:20.240ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f07556bb-dbbf-4461-be79-aaa83a4af1c8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39252025-04-30T06:38:20.240ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
39262025-04-30T06:38:20.242ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7342191-549e-4fa4-8d2d-aa4256a136c5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39272025-04-30T06:38:20.242ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
39282025-04-30T06:38:20.244ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccb61ad4-cdae-4473-9953-ebb2932b3be2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39292025-04-30T06:38:20.249ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
39302025-04-30T06:38:20.253ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "731bceb1-d2d2-471c-ad9d-25148cc491bc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39312025-04-30T06:38:20.253ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
39322025-04-30T06:38:20.255ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d4f8b0b-15b6-47b7-9587-5fe046cbffa6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39332025-04-30T06:38:20.256ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
39342025-04-30T06:38:20.260ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c22d99a-7a21-4a44-b813-139a5d2c46b6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39352025-04-30T06:38:20.260ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
39362025-04-30T06:38:20.263ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "811bab43-0207-4a04-a4fd-cab0bcc48676", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39372025-04-30T06:38:20.263ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
39382025-04-30T06:38:20.265ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fd18a80-e6cf-4d4a-866d-edf65a9601ce", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39392025-04-30T06:38:20.265ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
39402025-04-30T06:38:20.266ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39412025-04-30T06:38:20.266ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39422025-04-30T06:38:20.266ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39432025-04-30T06:38:20.266ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
39442025-04-30T06:38:20.267ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39452025-04-30T06:38:20.268ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d633a99-46a6-4c94-8b04-705c92628021", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39462025-04-30T06:38:20.268ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
39472025-04-30T06:38:20.270ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39482025-04-30T06:38:20.278ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39492025-04-30T06:38:20.279ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39502025-04-30T06:38:20.279ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
39512025-04-30T06:38:20.280ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39522025-04-30T06:38:20.280ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "251fa53c-91e0-4446-8e0f-5d254d4c60b9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39532025-04-30T06:38:20.280ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
39542025-04-30T06:38:20.283ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39552025-04-30T06:38:20.283ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
39562025-04-30T06:38:20.284ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39572025-04-30T06:38:20.285ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68576e6c-71dd-4e75-89ce-7b07737faa48", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39582025-04-30T06:38:20.285ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
39592025-04-30T06:38:20.288ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63e51b63-8af7-401e-87d9-df43a364ea5f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39602025-04-30T06:38:20.288ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
39612025-04-30T06:38:20.293ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d11b27a-3abc-4ccb-8439-6b351acd1622", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39622025-04-30T06:38:20.293ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
39632025-04-30T06:38:20.296ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d918e04d-73f1-4ae8-a01d-f949e9492877", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39642025-04-30T06:38:20.296ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
39652025-04-30T06:38:20.299ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff344f78-9452-4597-beaa-9450322be62c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39662025-04-30T06:38:20.299ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
39672025-04-30T06:38:20.305ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40b034e8-7c2b-476a-b4cb-8903da2b7d3d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39682025-04-30T06:38:20.305ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
39692025-04-30T06:38:20.312ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9765964a-0639-4869-a70c-6ef03d844874", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39702025-04-30T06:38:20.313ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
39712025-04-30T06:38:20.316ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cedd4fa-0ca2-4340-85f2-1a9fd44ef5a5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39722025-04-30T06:38:20.316ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
39732025-04-30T06:38:20.317ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
39742025-04-30T06:38:20.321ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d25345a-9780-48ec-978d-2806a5ee105b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39752025-04-30T06:38:20.321ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
39762025-04-30T06:38:20.324ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20a546e3-9f62-4a1e-a8ce-d6b97648736f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39772025-04-30T06:38:20.324ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
39782025-04-30T06:38:20.324ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d3a2ba1-7cca-4274-b810-5ed052b65a2c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
39792025-04-30T06:38:20.325ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
39802025-04-30T06:38:20.325ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39812025-04-30T06:38:20.325ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39822025-04-30T06:38:20.325ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39832025-04-30T06:38:20.338ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39842025-04-30T06:38:20.344ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39852025-04-30T06:38:20.351ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
39862025-04-30T06:38:20.351ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
39872025-04-30T06:38:20.351ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39882025-04-30T06:38:20.352ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39892025-04-30T06:38:20.352ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39902025-04-30T06:38:20.353ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39912025-04-30T06:38:20.356ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39922025-04-30T06:38:20.375ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
39932025-04-30T06:38:20.375ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39942025-04-30T06:38:20.375ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39952025-04-30T06:38:20.376ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39962025-04-30T06:38:20.377ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39972025-04-30T06:38:20.377ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39982025-04-30T06:38:20.378ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39992025-04-30T06:38:20.378ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
40002025-04-30T06:38:20.378ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40012025-04-30T06:38:20.403ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40022025-04-30T06:38:20.458ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40032025-04-30T06:38:20.458ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40042025-04-30T06:38:20.458ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
40052025-04-30T06:38:20.459ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40062025-04-30T06:38:20.545ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
40072025-04-30T06:38:20.545ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
40082025-04-30T06:38:20.547ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:50282 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
40092025-04-30T06:38:20.548ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:50282 req_id = f5cd605d-ef71-4647-9025-05640b169419 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40102025-04-30T06:38:20.549ZINFOtest_quotas (dropshot): request completed latency_us = 579 local_addr = [::1]:55655 method = GET remote_addr = [::1]:50282 req_id = f5cd605d-ef71-4647-9025-05640b169419 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40112025-04-30T06:38:20.549ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40122025-04-30T06:38:20.549ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
40132025-04-30T06:38:20.550ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40142025-04-30T06:38:20.550ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5cd605d-ef71-4647-9025-05640b169419", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40152025-04-30T06:38:20.552ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40162025-04-30T06:38:20.552ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40172025-04-30T06:38:20.552ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
40182025-04-30T06:38:20.553ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40192025-04-30T06:38:20.554ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40202025-04-30T06:38:20.554ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40212025-04-30T06:38:20.554ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40222025-04-30T06:38:20.554ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40232025-04-30T06:38:20.569ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40242025-04-30T06:38:20.576ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40252025-04-30T06:38:20.576ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40262025-04-30T06:38:20.576ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40272025-04-30T06:38:20.577ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40282025-04-30T06:38:20.621ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
40292025-04-30T06:38:20.629ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
40302025-04-30T06:38:20.630ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
40312025-04-30T06:38:20.631ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 528.478018ms iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
40322025-04-30T06:38:20.631ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
40332025-04-30T06:38:20.631ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40342025-04-30T06:38:20.631ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40352025-04-30T06:38:20.631ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40362025-04-30T06:38:20.631ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40372025-04-30T06:38:20.632ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40382025-04-30T06:38:20.640ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40392025-04-30T06:38:20.640ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40402025-04-30T06:38:20.640ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40412025-04-30T06:38:20.640ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40422025-04-30T06:38:20.640ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40432025-04-30T06:38:20.654ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
40442025-04-30T06:38:20.655ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
40452025-04-30T06:38:20.655ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f1 30 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .0..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56636 req_id = 6cf8525c-a67e-4bc1-b2d2-813ff4c43cbe
40462025-04-30T06:38:20.656ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61744,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56636 req_id = 6cf8525c-a67e-4bc1-b2d2-813ff4c43cbe
40472025-04-30T06:38:20.656ZDEBGtest_quotas (store): query key key = _mgs._tcp
40482025-04-30T06:38:20.656ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40492025-04-30T06:38:20.657ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56636 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6cf8525c-a67e-4bc1-b2d2-813ff4c43cbe
40502025-04-30T06:38:20.658ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3965, tv_nsec: 457233922 } })
40512025-04-30T06:38:20.658ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
40522025-04-30T06:38:20.659ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 30 6c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 0l..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61723 req_id = c2b08aa6-fccb-423c-b105-2363da582d9c
40532025-04-30T06:38:20.666ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12396,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61723 req_id = c2b08aa6-fccb-423c-b105-2363da582d9c
40542025-04-30T06:38:20.669ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40552025-04-30T06:38:20.669ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61723 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c2b08aa6-fccb-423c-b105-2363da582d9c
40562025-04-30T06:38:20.724ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
40572025-04-30T06:38:20.724ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40582025-04-30T06:38:20.724ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40592025-04-30T06:38:20.724ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40602025-04-30T06:38:20.725ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
40612025-04-30T06:38:20.725ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40622025-04-30T06:38:20.726ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 98 29 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .)..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49638 req_id = a0170930-8955-4355-9f67-e402daed4195
40632025-04-30T06:38:20.726ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38953,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49638 req_id = a0170930-8955-4355-9f67-e402daed4195
40642025-04-30T06:38:20.726ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
40652025-04-30T06:38:20.727ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38953, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49638 req_id = a0170930-8955-4355-9f67-e402daed4195
40662025-04-30T06:38:20.727ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
40672025-04-30T06:38:20.727ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
40682025-04-30T06:38:20.727ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
40692025-04-30T06:38:20.728ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40702025-04-30T06:38:20.737ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40712025-04-30T06:38:20.737ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40722025-04-30T06:38:20.737ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
40732025-04-30T06:38:20.738ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40742025-04-30T06:38:20.741ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "310a1888-da4d-4b44-9158-4a7beef7b77e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40752025-04-30T06:38:20.742ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
40762025-04-30T06:38:20.742ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40772025-04-30T06:38:20.742ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
40782025-04-30T06:38:20.743ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40792025-04-30T06:38:20.745ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19c35095-fb58-4c06-9031-540a2cb729a5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40802025-04-30T06:38:20.745ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
40812025-04-30T06:38:20.749ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c91c1df-982c-401f-a2c2-219d8a782b65", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40822025-04-30T06:38:20.749ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
40832025-04-30T06:38:20.752ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27dfdad3-3f20-42b4-ba50-905a64c3362a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40842025-04-30T06:38:20.753ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
40852025-04-30T06:38:20.755ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93f578f6-9bef-496e-8969-3d0cdd29a8e8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40862025-04-30T06:38:20.755ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
40872025-04-30T06:38:20.761ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c90e179-9b43-4365-b05a-568e4f153aca", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40882025-04-30T06:38:20.762ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
40892025-04-30T06:38:20.765ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "abdd0977-91d1-4765-a148-944d14a850f4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40902025-04-30T06:38:20.765ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
40912025-04-30T06:38:20.770ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
40922025-04-30T06:38:20.771ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51a554dd-52cf-4421-9533-c6c7fe320dda", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40932025-04-30T06:38:20.771ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
40942025-04-30T06:38:20.773ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e2181b0-082c-4fbd-8d29-b2917eb118bb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40952025-04-30T06:38:20.773ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
40962025-04-30T06:38:20.774ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81d0d5a8-7b12-402f-a176-6a9a57d35b15", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40972025-04-30T06:38:20.775ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
40982025-04-30T06:38:20.776ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7217c2a9-fa71-49eb-9ea6-e3e12697414b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
40992025-04-30T06:38:20.776ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
41002025-04-30T06:38:20.777ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dff01b0f-3e3d-43c0-9fed-dcbe4520420d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41012025-04-30T06:38:20.777ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
41022025-04-30T06:38:20.779ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1f4c17c-53dc-4f7c-a08b-73442a84eef5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41032025-04-30T06:38:20.780ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
41042025-04-30T06:38:20.784ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21c98f96-3171-4746-a5c2-52fb44a3022f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41052025-04-30T06:38:20.784ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
41062025-04-30T06:38:20.786ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bb8e4f6-b0c1-4888-b17d-1ed0c1609239", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41072025-04-30T06:38:20.786ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
41082025-04-30T06:38:20.788ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4f7e491-0a6d-4959-932c-2eca9ea3abf1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41092025-04-30T06:38:20.788ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
41102025-04-30T06:38:20.790ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61e0b050-57a7-4eaf-8b97-23ea1ea4c0f2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41112025-04-30T06:38:20.791ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
41122025-04-30T06:38:20.793ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d3e9ea2-8ee5-44de-a8b3-9540d835536e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41132025-04-30T06:38:20.793ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
41142025-04-30T06:38:20.795ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46b94a8f-32bc-42e4-86fb-59dbf13f865e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41152025-04-30T06:38:20.795ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
41162025-04-30T06:38:20.796ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20732337-efa2-414d-9210-4a1086ae44b6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41172025-04-30T06:38:20.796ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
41182025-04-30T06:38:20.797ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7023458-80be-49d6-aa3f-ff94b93557b3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41192025-04-30T06:38:20.797ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
41202025-04-30T06:38:20.803ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66119a9f-4458-40af-bc4b-41f861799dd9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41212025-04-30T06:38:20.803ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
41222025-04-30T06:38:20.804ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eefdd07-aa7c-49bc-9607-279d96db8546", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41232025-04-30T06:38:20.804ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
41242025-04-30T06:38:20.805ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dd85479-19dc-4318-8e67-8cc0faea89b7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41252025-04-30T06:38:20.807ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
41262025-04-30T06:38:20.809ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e635847-7c41-4a6f-96d1-fea4b87e3d00", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41272025-04-30T06:38:20.809ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
41282025-04-30T06:38:20.810ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4edbf741-3a86-4d22-bd55-ad097951dc72", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41292025-04-30T06:38:20.810ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
41302025-04-30T06:38:20.813ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bbc5706-963c-4446-87b8-af87db65a246", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41312025-04-30T06:38:20.813ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
41322025-04-30T06:38:20.815ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4924682-4574-46f3-b015-9283adde99c8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41332025-04-30T06:38:20.815ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
41342025-04-30T06:38:20.818ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "acb4baef-f0e8-421d-8c53-212ef444f05f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41352025-04-30T06:38:20.818ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
41362025-04-30T06:38:20.820ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8250ef2-be47-4cc2-8fee-5895b4c5ae6d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41372025-04-30T06:38:20.820ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
41382025-04-30T06:38:20.821ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "902a73fb-b4c5-410d-9c11-c6d75ac5b95b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41392025-04-30T06:38:20.822ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
41402025-04-30T06:38:20.822ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41412025-04-30T06:38:20.825ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41422025-04-30T06:38:20.825ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41432025-04-30T06:38:20.825ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
41442025-04-30T06:38:20.826ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41452025-04-30T06:38:20.828ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7749046f-ab75-490d-9fba-6bf18d30b342", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41462025-04-30T06:38:20.828ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
41472025-04-30T06:38:20.832ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41482025-04-30T06:38:20.839ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41492025-04-30T06:38:20.840ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41502025-04-30T06:38:20.840ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
41512025-04-30T06:38:20.840ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41522025-04-30T06:38:20.842ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6678b475-e846-46b2-81e3-51ec0617bd3b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41532025-04-30T06:38:20.842ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
41542025-04-30T06:38:20.843ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41552025-04-30T06:38:20.843ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
41562025-04-30T06:38:20.844ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41572025-04-30T06:38:20.844ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c871ceb-f119-4db8-a1b1-4789b18e8977", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41582025-04-30T06:38:20.844ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
41592025-04-30T06:38:20.846ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d68206dc-23e0-44b2-8445-dabb0538930a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41602025-04-30T06:38:20.846ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
41612025-04-30T06:38:20.849ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e35496d0-0799-4c58-ba8c-edfd008d7582", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41622025-04-30T06:38:20.849ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
41632025-04-30T06:38:20.850ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41642025-04-30T06:38:20.850ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41652025-04-30T06:38:20.853ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41662025-04-30T06:38:20.854ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41672025-04-30T06:38:20.854ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41682025-04-30T06:38:20.854ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41692025-04-30T06:38:20.857ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3a7da12-aba4-4b2d-b23d-b5d17d902c21", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41702025-04-30T06:38:20.858ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
41712025-04-30T06:38:20.860ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c7c596c-355f-444c-a611-e7cf06e26a6d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41722025-04-30T06:38:20.860ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
41732025-04-30T06:38:20.862ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38cd2685-1589-4cc1-b07b-50993e29f642", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41742025-04-30T06:38:20.862ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
41752025-04-30T06:38:20.864ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12c1cc73-0738-4fcd-a526-450d65d3f1cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41762025-04-30T06:38:20.864ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
41772025-04-30T06:38:20.866ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46d6af5f-c588-46a0-9ff2-a381b40defb8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41782025-04-30T06:38:20.866ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
41792025-04-30T06:38:20.868ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e6607c0-0f1b-4773-bb44-ced7de942f96", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41802025-04-30T06:38:20.868ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
41812025-04-30T06:38:20.869ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc16353b-0f35-4a48-ac79-26b0eb39a964", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41822025-04-30T06:38:20.869ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
41832025-04-30T06:38:20.871ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e4cda53-5ae2-4e08-8f24-a660a07a2d8d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41842025-04-30T06:38:20.872ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
41852025-04-30T06:38:20.873ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
41862025-04-30T06:38:20.875ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa8e13cc-6ae8-4f8d-8890-67a14c3678ba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
41872025-04-30T06:38:20.875ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
41882025-04-30T06:38:20.875ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41892025-04-30T06:38:20.875ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41902025-04-30T06:38:20.875ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41912025-04-30T06:38:20.876ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41922025-04-30T06:38:20.879ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41932025-04-30T06:38:20.885ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
41942025-04-30T06:38:20.885ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
41952025-04-30T06:38:20.885ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41962025-04-30T06:38:20.885ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41972025-04-30T06:38:20.885ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41982025-04-30T06:38:20.885ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41992025-04-30T06:38:20.888ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42002025-04-30T06:38:20.900ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42012025-04-30T06:38:20.901ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42022025-04-30T06:38:20.901ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42032025-04-30T06:38:20.902ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42042025-04-30T06:38:20.947ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
42052025-04-30T06:38:20.947ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
42062025-04-30T06:38:20.947ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42072025-04-30T06:38:20.947ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42082025-04-30T06:38:20.947ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42092025-04-30T06:38:20.947ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
42102025-04-30T06:38:20.948ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42112025-04-30T06:38:20.948ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:62087 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
42122025-04-30T06:38:20.949ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:62087 req_id = 220c1310-25b0-4bf0-a24e-7e779e2e752a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42132025-04-30T06:38:20.950ZINFOtest_quotas (dropshot): request completed latency_us = 460 local_addr = [::1]:55655 method = GET remote_addr = [::1]:62087 req_id = 220c1310-25b0-4bf0-a24e-7e779e2e752a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42142025-04-30T06:38:20.951ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "220c1310-25b0-4bf0-a24e-7e779e2e752a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:19 GMT"} })
42152025-04-30T06:38:20.952ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42162025-04-30T06:38:20.952ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42172025-04-30T06:38:20.952ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
42182025-04-30T06:38:20.954ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42192025-04-30T06:38:20.954ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42202025-04-30T06:38:20.954ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42212025-04-30T06:38:20.954ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42222025-04-30T06:38:20.954ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42232025-04-30T06:38:20.955ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42242025-04-30T06:38:20.960ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42252025-04-30T06:38:20.960ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42262025-04-30T06:38:20.960ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
42272025-04-30T06:38:20.960ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42282025-04-30T06:38:20.964ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42292025-04-30T06:38:20.967ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42302025-04-30T06:38:20.967ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42312025-04-30T06:38:20.967ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42322025-04-30T06:38:20.968ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42332025-04-30T06:38:20.973ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42342025-04-30T06:38:20.974ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
42352025-04-30T06:38:20.974ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42362025-04-30T06:38:21.075ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
42372025-04-30T06:38:21.075ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
42382025-04-30T06:38:21.076ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 444.821327ms iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
42392025-04-30T06:38:21.076ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
42402025-04-30T06:38:21.076ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42412025-04-30T06:38:21.077ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42422025-04-30T06:38:21.078ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42432025-04-30T06:38:21.078ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42442025-04-30T06:38:21.080ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42452025-04-30T06:38:21.082ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42462025-04-30T06:38:21.101ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42472025-04-30T06:38:21.111ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42482025-04-30T06:38:21.111ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42492025-04-30T06:38:21.112ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42502025-04-30T06:38:21.120ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
42512025-04-30T06:38:21.120ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
42522025-04-30T06:38:21.122ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42532025-04-30T06:38:21.122ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42542025-04-30T06:38:21.122ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
42552025-04-30T06:38:21.123ZDEBGtest_quotas (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
42562025-04-30T06:38:21.123ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
42572025-04-30T06:38:21.124ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
42582025-04-30T06:38:21.124ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fb f2 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52597 req_id = d0da68ea-41d7-4fa4-ae29-5030e1ac4055
42592025-04-30T06:38:21.124ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64498,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52597 req_id = d0da68ea-41d7-4fa4-ae29-5030e1ac4055
42602025-04-30T06:38:21.124ZDEBGtest_quotas (store): query key key = _mgs._tcp
42612025-04-30T06:38:21.124ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42622025-04-30T06:38:21.125ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52597 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d0da68ea-41d7-4fa4-ae29-5030e1ac4055
42632025-04-30T06:38:21.125ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3965, tv_nsec: 924586390 } })
42642025-04-30T06:38:21.125ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
42652025-04-30T06:38:21.126ZTRACtest_quotas (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
42662025-04-30T06:38:21.126ZDEBGtest_quotas (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:38:21.123649856Z } timeseries_name = integration_target:integration_metric
42672025-04-30T06:38:21.127ZDEBGtest_quotas (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
42682025-04-30T06:38:21.127ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
42692025-04-30T06:38:21.127ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e9 87 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62191 req_id = 489d33d2-e20a-4419-acd6-c28c3d354ede
42702025-04-30T06:38:21.127ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59783,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62191 req_id = 489d33d2-e20a-4419-acd6-c28c3d354ede
42712025-04-30T06:38:21.127ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42722025-04-30T06:38:21.127ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62191 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 489d33d2-e20a-4419-acd6-c28c3d354ede
42732025-04-30T06:38:21.210ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
42742025-04-30T06:38:21.211ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42752025-04-30T06:38:21.211ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42762025-04-30T06:38:21.211ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42772025-04-30T06:38:21.211ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
42782025-04-30T06:38:21.224ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42792025-04-30T06:38:21.231ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
42802025-04-30T06:38:21.232ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 18 d0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49960 req_id = e410b846-fb3b-4ce7-bf57-13f0bfdc8731
42812025-04-30T06:38:21.232ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6352,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49960 req_id = e410b846-fb3b-4ce7-bf57-13f0bfdc8731
42822025-04-30T06:38:21.233ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
42832025-04-30T06:38:21.233ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6352, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49960 req_id = e410b846-fb3b-4ce7-bf57-13f0bfdc8731
42842025-04-30T06:38:21.233ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
42852025-04-30T06:38:21.252ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
42862025-04-30T06:38:21.252ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
42872025-04-30T06:38:21.252ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
42882025-04-30T06:38:21.252ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42892025-04-30T06:38:21.259ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42902025-04-30T06:38:21.259ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42912025-04-30T06:38:21.259ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
42922025-04-30T06:38:21.259ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42932025-04-30T06:38:21.260ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
42942025-04-30T06:38:21.260ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
42952025-04-30T06:38:21.276ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
42962025-04-30T06:38:21.276ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42972025-04-30T06:38:21.276ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42982025-04-30T06:38:21.276ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
42992025-04-30T06:38:21.297ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43002025-04-30T06:38:21.301ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3a81ba0-7339-4958-89b4-92f870958adc", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43012025-04-30T06:38:21.301ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
43022025-04-30T06:38:21.305ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0217e693-1170-4309-9ad2-2bcbe03ecd8a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43032025-04-30T06:38:21.305ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
43042025-04-30T06:38:21.312ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88e083a4-b801-4935-94ae-34d26fda6600", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43052025-04-30T06:38:21.313ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
43062025-04-30T06:38:21.322ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52aaff6b-8c7f-4d60-9d23-044fac074636", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43072025-04-30T06:38:21.325ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
43082025-04-30T06:38:21.330ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "756d3c02-f951-454b-9a0e-64972d2eccd7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43092025-04-30T06:38:21.330ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
43102025-04-30T06:38:21.334ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "774067a0-335b-461f-8904-0c516ff02c33", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43112025-04-30T06:38:21.334ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
43122025-04-30T06:38:21.337ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b12d49f3-967d-4d8c-af1e-e5885eb97cfb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43132025-04-30T06:38:21.337ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
43142025-04-30T06:38:21.340ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ed7bcba-4e7a-4540-a9f1-ceaa92f254ec", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43152025-04-30T06:38:21.340ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
43162025-04-30T06:38:21.341ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
43172025-04-30T06:38:21.367ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffe5a79d-eff5-436a-95fa-1738d4fce57e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43182025-04-30T06:38:21.367ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
43192025-04-30T06:38:21.393ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43202025-04-30T06:38:21.393ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43212025-04-30T06:38:21.393ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43222025-04-30T06:38:21.393ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
43232025-04-30T06:38:21.396ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43242025-04-30T06:38:21.398ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd66c89b-5639-46a8-8a0d-6e1a30cbaa8b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43252025-04-30T06:38:21.398ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
43262025-04-30T06:38:21.401ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43272025-04-30T06:38:21.409ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43282025-04-30T06:38:21.409ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43292025-04-30T06:38:21.410ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
43302025-04-30T06:38:21.410ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43312025-04-30T06:38:21.410ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f07f605e-2af9-4778-a8c8-876bca9697bb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43322025-04-30T06:38:21.410ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
43332025-04-30T06:38:21.413ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f99eacf-04bb-4ad2-a8e9-ce29110d0995", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43342025-04-30T06:38:21.413ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
43352025-04-30T06:38:21.414ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb2b5ce8-57b4-400d-9c66-189b19063706", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43362025-04-30T06:38:21.415ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
43372025-04-30T06:38:21.415ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43382025-04-30T06:38:21.415ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
43392025-04-30T06:38:21.415ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43402025-04-30T06:38:21.419ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d048a18-a6b4-4131-bc90-586f8884e2e6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43412025-04-30T06:38:21.419ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
43422025-04-30T06:38:21.422ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ef8e86c-e700-444d-bd18-8089228f1db5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43432025-04-30T06:38:21.422ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
43442025-04-30T06:38:21.473ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20926661-f885-4dd1-8742-33b255dfd97c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43452025-04-30T06:38:21.473ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
43462025-04-30T06:38:21.483ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6dc580c-ee61-478d-bd25-55c4d5496761", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43472025-04-30T06:38:21.503ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
43482025-04-30T06:38:21.515ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cecba185-05eb-4cdc-9f29-b9c4dcc52556", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43492025-04-30T06:38:21.515ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
43502025-04-30T06:38:21.518ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c659df52-5581-4c2a-84c9-c035ae3d0e3e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43512025-04-30T06:38:21.518ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
43522025-04-30T06:38:21.529ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
43532025-04-30T06:38:21.529ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a7ddb4e-ff50-4689-94fb-73068edefb50", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43542025-04-30T06:38:21.530ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
43552025-04-30T06:38:21.531ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e212cba-afb7-44ab-b2f0-0dd32741f6f4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43562025-04-30T06:38:21.531ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
43572025-04-30T06:38:21.535ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f851583-f14d-4f78-8ca3-495a6bf5ebd5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43582025-04-30T06:38:21.535ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
43592025-04-30T06:38:21.537ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b734048a-2f13-45b4-9593-f708131c31f1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43602025-04-30T06:38:21.537ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
43612025-04-30T06:38:21.538ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98947950-9d05-430b-8df2-e70277efa4dd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43622025-04-30T06:38:21.538ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
43632025-04-30T06:38:21.547ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb86a197-a2f4-4a53-b462-e53e3762ce7e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43642025-04-30T06:38:21.547ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
43652025-04-30T06:38:21.554ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "692f8695-72c2-4136-a95a-3090d522f427", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43662025-04-30T06:38:21.555ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
43672025-04-30T06:38:21.557ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b1aff28-d4a3-43f1-9ac1-659ace1c870c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43682025-04-30T06:38:21.557ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
43692025-04-30T06:38:21.559ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a054579a-af19-483b-a05e-eb53b455b3b5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43702025-04-30T06:38:21.559ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
43712025-04-30T06:38:21.601ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43722025-04-30T06:38:21.601ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43732025-04-30T06:38:21.601ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43742025-04-30T06:38:21.601ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
43752025-04-30T06:38:21.602ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43762025-04-30T06:38:21.605ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43772025-04-30T06:38:21.611ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43782025-04-30T06:38:21.611ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43792025-04-30T06:38:21.611ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
43802025-04-30T06:38:21.611ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43812025-04-30T06:38:21.612ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fce62d8-18a7-4a09-a1e3-5d7ddb043949", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43822025-04-30T06:38:21.612ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
43832025-04-30T06:38:21.615ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43842025-04-30T06:38:21.615ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
43852025-04-30T06:38:21.615ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43862025-04-30T06:38:21.619ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d6f3cd5-59b8-4354-9851-7a38081f46ef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43872025-04-30T06:38:21.619ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
43882025-04-30T06:38:21.620ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb3b7878-1235-41a9-983e-4b46b53dfdec", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43892025-04-30T06:38:21.622ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
43902025-04-30T06:38:21.642ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c30b13c7-b257-4c60-bcdc-c38dbb9104ac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43912025-04-30T06:38:21.643ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
43922025-04-30T06:38:21.644ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "efa52e24-447a-48a2-9da2-b861e51b589b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43932025-04-30T06:38:21.645ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
43942025-04-30T06:38:21.646ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e77c3ff8-7982-44b1-ba40-ceb9a611d86a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43952025-04-30T06:38:21.655ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
43962025-04-30T06:38:21.693ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dea4e623-a420-44fb-a7cc-906f3ba6dab4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43972025-04-30T06:38:21.693ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
43982025-04-30T06:38:21.698ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84ecc1ef-c305-40d3-a895-72163c54d545", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
43992025-04-30T06:38:21.698ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
44002025-04-30T06:38:21.702ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2acc30e2-c793-4278-bd92-45c4d1b5ae4e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44012025-04-30T06:38:21.703ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
44022025-04-30T06:38:21.712ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2ee4850-13c8-4a95-b60a-e0f051798f04", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44032025-04-30T06:38:21.713ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
44042025-04-30T06:38:21.757ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
44052025-04-30T06:38:21.759ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
44062025-04-30T06:38:21.759ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53d3c190-f0e0-45bd-876c-35c1a64567f9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44072025-04-30T06:38:21.759ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
44082025-04-30T06:38:21.762ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44092025-04-30T06:38:21.764ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 2ea99027-e9e5-4f4a-8aaf-94ebab867c38 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44102025-04-30T06:38:21.765ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 2ea99027-e9e5-4f4a-8aaf-94ebab867c38 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44112025-04-30T06:38:21.765ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 2ea99027-e9e5-4f4a-8aaf-94ebab867c38 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44122025-04-30T06:38:21.768ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 2ea99027-e9e5-4f4a-8aaf-94ebab867c38 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44132025-04-30T06:38:21.770ZINFOtest_quotas (dropshot_internal): request completed latency_us = 5647 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 2ea99027-e9e5-4f4a-8aaf-94ebab867c38 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
44142025-04-30T06:38:21.770ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ea99027-e9e5-4f4a-8aaf-94ebab867c38", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44152025-04-30T06:38:21.770ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44162025-04-30T06:38:21.771ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 05be5c56-aaa2-43bc-ae9e-1a3c816a1b23 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44172025-04-30T06:38:21.771ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 05be5c56-aaa2-43bc-ae9e-1a3c816a1b23 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44182025-04-30T06:38:21.772ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 05be5c56-aaa2-43bc-ae9e-1a3c816a1b23 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44192025-04-30T06:38:21.772ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 05be5c56-aaa2-43bc-ae9e-1a3c816a1b23 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44202025-04-30T06:38:21.773ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74e6a429-c330-461f-acfa-3d9211d24281", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44212025-04-30T06:38:21.774ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
44222025-04-30T06:38:21.775ZINFOtest_quotas (dropshot_internal): request completed latency_us = 4511 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 05be5c56-aaa2-43bc-ae9e-1a3c816a1b23 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
44232025-04-30T06:38:21.802ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "05be5c56-aaa2-43bc-ae9e-1a3c816a1b23", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44242025-04-30T06:38:21.802ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
44252025-04-30T06:38:21.802ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44262025-04-30T06:38:21.802ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
44272025-04-30T06:38:21.811ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9bdf8f0-7a45-4560-b90d-869d14f0bba1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44282025-04-30T06:38:21.811ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
44292025-04-30T06:38:21.811ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44302025-04-30T06:38:21.811ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44312025-04-30T06:38:21.811ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44322025-04-30T06:38:21.812ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
44332025-04-30T06:38:21.812ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44342025-04-30T06:38:21.814ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "423a80cf-5b17-41fc-81e2-a110ae646ef9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44352025-04-30T06:38:21.814ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
44362025-04-30T06:38:21.816ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44372025-04-30T06:38:21.830ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44382025-04-30T06:38:21.831ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44392025-04-30T06:38:21.831ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
44402025-04-30T06:38:21.841ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44412025-04-30T06:38:21.846ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bab27469-08f9-44c4-bd7f-738834e919b7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44422025-04-30T06:38:21.846ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
44432025-04-30T06:38:21.847ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44442025-04-30T06:38:21.847ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
44452025-04-30T06:38:21.861ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44462025-04-30T06:38:21.861ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44472025-04-30T06:38:21.861ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a2a56da-44ad-4144-be7e-908767384db4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44482025-04-30T06:38:21.861ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
44492025-04-30T06:38:21.861ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44502025-04-30T06:38:21.861ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44512025-04-30T06:38:21.861ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44522025-04-30T06:38:21.862ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44532025-04-30T06:38:21.862ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44542025-04-30T06:38:21.862ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "551ac374-c836-44d0-a242-c6eee269852f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:20 GMT"} })
44552025-04-30T06:38:21.862ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44562025-04-30T06:38:21.862ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44572025-04-30T06:38:21.862ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44582025-04-30T06:38:21.862ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44592025-04-30T06:38:21.862ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44602025-04-30T06:38:21.866ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44612025-04-30T06:38:21.908ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44622025-04-30T06:38:21.918ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44632025-04-30T06:38:21.925ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44642025-04-30T06:38:21.925ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44652025-04-30T06:38:21.925ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44662025-04-30T06:38:21.926ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44672025-04-30T06:38:21.947ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44682025-04-30T06:38:21.951ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44692025-04-30T06:38:21.951ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44702025-04-30T06:38:21.951ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44712025-04-30T06:38:21.951ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44722025-04-30T06:38:22.131ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
44732025-04-30T06:38:22.131ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
44742025-04-30T06:38:22.132ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:44601 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
44752025-04-30T06:38:22.132ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44762025-04-30T06:38:22.132ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44772025-04-30T06:38:22.133ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
44782025-04-30T06:38:22.146ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
44792025-04-30T06:38:22.149ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:44601 req_id = 933c6930-d406-4dc8-9265-250c90bf41b6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44802025-04-30T06:38:22.160ZINFOtest_quotas (dropshot): request completed latency_us = 10645 local_addr = [::1]:55655 method = GET remote_addr = [::1]:44601 req_id = 933c6930-d406-4dc8-9265-250c90bf41b6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44812025-04-30T06:38:22.168ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "933c6930-d406-4dc8-9265-250c90bf41b6", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
44822025-04-30T06:38:22.174ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44832025-04-30T06:38:22.174ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44842025-04-30T06:38:22.174ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
44852025-04-30T06:38:22.178ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44862025-04-30T06:38:22.179ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44872025-04-30T06:38:22.179ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44882025-04-30T06:38:22.179ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44892025-04-30T06:38:22.180ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44902025-04-30T06:38:22.183ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
44912025-04-30T06:38:22.183ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
44922025-04-30T06:38:22.205ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44932025-04-30T06:38:22.234ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44942025-04-30T06:38:22.234ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44952025-04-30T06:38:22.234ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44962025-04-30T06:38:22.235ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44972025-04-30T06:38:22.239ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
44982025-04-30T06:38:22.240ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44992025-04-30T06:38:22.240ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
45002025-04-30T06:38:22.249ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
45012025-04-30T06:38:22.301ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45022025-04-30T06:38:22.301ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45032025-04-30T06:38:22.301ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45042025-04-30T06:38:22.301ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
45052025-04-30T06:38:22.302ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45062025-04-30T06:38:22.319ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45072025-04-30T06:38:22.336ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45082025-04-30T06:38:22.339ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45092025-04-30T06:38:22.339ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
45102025-04-30T06:38:22.341ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45112025-04-30T06:38:22.352ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
45122025-04-30T06:38:22.370ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
45132025-04-30T06:38:22.370ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.29409532s iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
45142025-04-30T06:38:22.370ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
45152025-04-30T06:38:22.370ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
45162025-04-30T06:38:22.370ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45172025-04-30T06:38:22.370ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45182025-04-30T06:38:22.371ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45192025-04-30T06:38:22.372ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45202025-04-30T06:38:22.373ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45212025-04-30T06:38:22.374ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
45222025-04-30T06:38:22.375ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45232025-04-30T06:38:22.375ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45242025-04-30T06:38:22.379ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
45252025-04-30T06:38:22.379ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45262025-04-30T06:38:22.380ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45272025-04-30T06:38:22.380ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45282025-04-30T06:38:22.384ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
45292025-04-30T06:38:22.384ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
45302025-04-30T06:38:22.386ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 87 25 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .%..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52286 req_id = 3883f66d-847c-4202-860b-2bc3fd30dcb3
45312025-04-30T06:38:22.403ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34597,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52286 req_id = 3883f66d-847c-4202-860b-2bc3fd30dcb3
45322025-04-30T06:38:22.404ZDEBGtest_quotas (store): query key key = _mgs._tcp
45332025-04-30T06:38:22.404ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45342025-04-30T06:38:22.404ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52286 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3883f66d-847c-4202-860b-2bc3fd30dcb3
45352025-04-30T06:38:22.405ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3967, tv_nsec: 203932343 } })
45362025-04-30T06:38:22.405ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
45372025-04-30T06:38:22.405ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6b a5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e k...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55233 req_id = 55cd0f2b-ff85-4933-84a5-fdba5ec5811d
45382025-04-30T06:38:22.405ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27557,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55233 req_id = 55cd0f2b-ff85-4933-84a5-fdba5ec5811d
45392025-04-30T06:38:22.405ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45402025-04-30T06:38:22.405ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55233 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 55cd0f2b-ff85-4933-84a5-fdba5ec5811d
45412025-04-30T06:38:22.555ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
45422025-04-30T06:38:22.557ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b7 db 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56904 req_id = 5bc10b2b-3e73-4331-a9e3-066060d37579
45432025-04-30T06:38:22.557ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47067,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56904 req_id = 5bc10b2b-3e73-4331-a9e3-066060d37579
45442025-04-30T06:38:22.557ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
45452025-04-30T06:38:22.557ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47067, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56904 req_id = 5bc10b2b-3e73-4331-a9e3-066060d37579
45462025-04-30T06:38:22.557ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
45472025-04-30T06:38:22.557ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
45482025-04-30T06:38:22.557ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
45492025-04-30T06:38:22.560ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bac8f5d-605e-43b5-8a64-f76e7b756402", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45502025-04-30T06:38:22.560ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
45512025-04-30T06:38:22.569ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d55ffaf7-b83a-4135-adcc-7f575f74a185", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45522025-04-30T06:38:22.578ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
45532025-04-30T06:38:22.606ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5768ee61-1498-4e6c-991a-1f9f95c6b981", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45542025-04-30T06:38:22.606ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
45552025-04-30T06:38:22.607ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
45562025-04-30T06:38:22.611ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6cef6ae-c9f1-41ca-86ea-745fcdca4e30", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45572025-04-30T06:38:22.612ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
45582025-04-30T06:38:22.638ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd4ccdfc-7858-4716-9192-00e1c1028151", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45592025-04-30T06:38:22.638ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
45602025-04-30T06:38:22.640ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04d0134a-c650-4caf-bf84-83a510bda1ce", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45612025-04-30T06:38:22.640ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
45622025-04-30T06:38:22.642ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21db72d4-a75e-4463-9629-25648bb9d240", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45632025-04-30T06:38:22.642ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
45642025-04-30T06:38:22.645ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5600f49a-f320-4c4a-a893-ddd7d740e2b0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45652025-04-30T06:38:22.645ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
45662025-04-30T06:38:22.646ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71aa9a6e-9784-40eb-9262-a350f9b62e08", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45672025-04-30T06:38:22.646ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
45682025-04-30T06:38:22.647ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b289079d-6ec4-41cf-b420-189bd65dab71", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45692025-04-30T06:38:22.647ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
45702025-04-30T06:38:22.648ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a321286-06d1-4bb5-9b8c-9e1b12b3f32f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45712025-04-30T06:38:22.648ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
45722025-04-30T06:38:22.656ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3917b8e8-8dba-4861-841a-92b565a72679", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45732025-04-30T06:38:22.656ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
45742025-04-30T06:38:22.660ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45752025-04-30T06:38:22.660ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45762025-04-30T06:38:22.660ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45772025-04-30T06:38:22.660ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
45782025-04-30T06:38:22.662ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45792025-04-30T06:38:22.669ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45802025-04-30T06:38:22.679ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45812025-04-30T06:38:22.679ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45822025-04-30T06:38:22.680ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
45832025-04-30T06:38:22.680ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45842025-04-30T06:38:22.680ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "164805d7-7495-4aa5-bc06-f38290d8874e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45852025-04-30T06:38:22.681ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
45862025-04-30T06:38:22.683ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45872025-04-30T06:38:22.683ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
45882025-04-30T06:38:22.686ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45892025-04-30T06:38:22.689ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a22f7115-362f-4501-9cd5-265b2f4e3fcc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45902025-04-30T06:38:22.689ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
45912025-04-30T06:38:22.695ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d18a2554-5e9f-4aa1-b379-952f5ea3e47e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45922025-04-30T06:38:22.696ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
45932025-04-30T06:38:22.701ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "187160f0-7321-4658-96f4-4fee6bcb880c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45942025-04-30T06:38:22.701ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
45952025-04-30T06:38:22.709ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ea87008-503d-44ff-a869-0a7a532740e0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45962025-04-30T06:38:22.709ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
45972025-04-30T06:38:22.720ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd075e15-87d9-4ce9-99d5-e53bd466a9e4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
45982025-04-30T06:38:22.721ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
45992025-04-30T06:38:22.722ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
46002025-04-30T06:38:22.724ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "334a445a-2fab-43f8-988d-dc7978bde0f5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46012025-04-30T06:38:22.724ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
46022025-04-30T06:38:22.725ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b25a6abb-2cf6-494d-9749-cd4e8a211d3e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46032025-04-30T06:38:22.725ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
46042025-04-30T06:38:22.726ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad0c7feb-f492-4458-92bc-acbe90add33a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46052025-04-30T06:38:22.726ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
46062025-04-30T06:38:22.728ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54d4ea6c-0135-4b12-b48a-a3072e1964be", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46072025-04-30T06:38:22.728ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
46082025-04-30T06:38:22.729ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a04c7f9e-0ecc-4cbf-9c6e-6657ce440a0b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46092025-04-30T06:38:22.729ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
46102025-04-30T06:38:22.730ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1a21da3-6ffe-44c1-8ce4-53cb759bb7a5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46112025-04-30T06:38:22.731ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
46122025-04-30T06:38:22.733ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bb47941-30bc-4565-831f-a8ee21b7ec19", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46132025-04-30T06:38:22.733ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
46142025-04-30T06:38:22.735ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33a9fed0-6939-408b-892e-4010ef178557", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46152025-04-30T06:38:22.735ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
46162025-04-30T06:38:22.739ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef2f48c0-b3b3-451c-9c99-b7a2cc15b476", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46172025-04-30T06:38:22.739ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
46182025-04-30T06:38:22.741ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d3fa411-3fd6-41dc-acde-775668ac40bd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46192025-04-30T06:38:22.741ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
46202025-04-30T06:38:22.746ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37f6ddb9-f29b-4abe-9de1-0448e14d627d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46212025-04-30T06:38:22.746ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
46222025-04-30T06:38:22.750ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d07c334-e3d7-4047-97a8-36fb2195be82", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46232025-04-30T06:38:22.751ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
46242025-04-30T06:38:22.752ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69d4efce-9479-4597-b792-0ff7f3f37e8a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46252025-04-30T06:38:22.752ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
46262025-04-30T06:38:22.753ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c4bac8d-3b6f-4320-ad45-bc597515bdb4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46272025-04-30T06:38:22.753ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
46282025-04-30T06:38:22.754ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "636e5db8-6ff9-4972-9004-087fc093ad06", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46292025-04-30T06:38:22.755ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
46302025-04-30T06:38:22.756ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ea39273-7ffd-4773-aa98-749373e8d9d2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46312025-04-30T06:38:22.757ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
46322025-04-30T06:38:22.759ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4643def4-d924-4f95-af33-b136cbd080c4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46332025-04-30T06:38:22.759ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
46342025-04-30T06:38:22.762ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e66ac9c2-1f37-4399-9ace-f258dc9aa350", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46352025-04-30T06:38:22.762ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
46362025-04-30T06:38:22.768ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bb54422-e020-4148-a488-dd3a4629a862", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46372025-04-30T06:38:22.769ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
46382025-04-30T06:38:22.771ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95eb9184-d87a-4c72-9412-fc69ffe1076e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46392025-04-30T06:38:22.772ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
46402025-04-30T06:38:22.773ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46412025-04-30T06:38:22.773ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46422025-04-30T06:38:22.773ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46432025-04-30T06:38:22.773ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
46442025-04-30T06:38:22.774ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46452025-04-30T06:38:22.775ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f5e33af-3181-4e30-aed5-faf9630cb207", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46462025-04-30T06:38:22.777ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
46472025-04-30T06:38:22.777ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46482025-04-30T06:38:22.787ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46492025-04-30T06:38:22.787ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46502025-04-30T06:38:22.787ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
46512025-04-30T06:38:22.788ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46522025-04-30T06:38:22.836ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46532025-04-30T06:38:22.836ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
46542025-04-30T06:38:22.837ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46552025-04-30T06:38:22.839ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0e11e10-f58f-4a13-bd2e-5425d6b947ab", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46562025-04-30T06:38:22.839ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
46572025-04-30T06:38:22.841ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5418248-5761-4835-9d15-b6d577d25da5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46582025-04-30T06:38:22.841ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
46592025-04-30T06:38:22.846ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "867f61e0-881c-4d17-ab95-4a62dd5d625a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46602025-04-30T06:38:22.846ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
46612025-04-30T06:38:22.848ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab9e6286-4d67-4d25-b566-c7ea54080292", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46622025-04-30T06:38:22.848ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
46632025-04-30T06:38:22.850ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46642025-04-30T06:38:22.850ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46652025-04-30T06:38:22.852ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b48bd6a-0bf3-44dd-94e7-8bf87404444c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46662025-04-30T06:38:22.852ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
46672025-04-30T06:38:22.854ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46682025-04-30T06:38:22.854ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46692025-04-30T06:38:22.855ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46702025-04-30T06:38:22.855ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46712025-04-30T06:38:22.856ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "696821d0-1ec4-4425-babf-3da5dad91ecb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46722025-04-30T06:38:22.856ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46732025-04-30T06:38:22.856ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46742025-04-30T06:38:22.857ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46752025-04-30T06:38:22.857ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46762025-04-30T06:38:22.858ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46772025-04-30T06:38:22.863ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46782025-04-30T06:38:22.868ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46792025-04-30T06:38:22.868ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46802025-04-30T06:38:22.868ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46812025-04-30T06:38:22.868ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46822025-04-30T06:38:22.869ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46832025-04-30T06:38:22.869ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46842025-04-30T06:38:22.877ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46852025-04-30T06:38:22.883ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46862025-04-30T06:38:22.883ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46872025-04-30T06:38:22.883ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46882025-04-30T06:38:22.884ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46892025-04-30T06:38:22.939ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
46902025-04-30T06:38:22.939ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
46912025-04-30T06:38:22.940ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:57749 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
46922025-04-30T06:38:22.956ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:57749 req_id = adeb49a9-d1be-4320-ae1d-4a4625e77bd7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46932025-04-30T06:38:22.957ZINFOtest_quotas (dropshot): request completed latency_us = 757 local_addr = [::1]:55655 method = GET remote_addr = [::1]:57749 req_id = adeb49a9-d1be-4320-ae1d-4a4625e77bd7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46942025-04-30T06:38:22.959ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "adeb49a9-d1be-4320-ae1d-4a4625e77bd7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:22 GMT"} })
46952025-04-30T06:38:22.961ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46962025-04-30T06:38:22.961ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46972025-04-30T06:38:22.961ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
46982025-04-30T06:38:22.962ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46992025-04-30T06:38:22.962ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47002025-04-30T06:38:22.962ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47012025-04-30T06:38:22.962ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47022025-04-30T06:38:22.963ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47032025-04-30T06:38:22.963ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
47042025-04-30T06:38:22.968ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47052025-04-30T06:38:22.975ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47062025-04-30T06:38:22.975ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47072025-04-30T06:38:22.975ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47082025-04-30T06:38:22.975ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47092025-04-30T06:38:23.021ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47102025-04-30T06:38:23.021ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47112025-04-30T06:38:23.021ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47122025-04-30T06:38:23.021ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
47132025-04-30T06:38:23.022ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47142025-04-30T06:38:23.039ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47152025-04-30T06:38:23.048ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47162025-04-30T06:38:23.048ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47172025-04-30T06:38:23.048ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
47182025-04-30T06:38:23.048ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47192025-04-30T06:38:23.103ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47202025-04-30T06:38:23.103ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47212025-04-30T06:38:23.103ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
47222025-04-30T06:38:23.104ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
47232025-04-30T06:38:23.113ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
47242025-04-30T06:38:23.113ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
47252025-04-30T06:38:23.134ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
47262025-04-30T06:38:23.134ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47272025-04-30T06:38:23.140ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
47282025-04-30T06:38:23.140ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
47292025-04-30T06:38:23.140ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 769.926789ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
47302025-04-30T06:38:23.140ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
47312025-04-30T06:38:23.140ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
47322025-04-30T06:38:23.140ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47332025-04-30T06:38:23.140ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47342025-04-30T06:38:23.140ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47352025-04-30T06:38:23.141ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47362025-04-30T06:38:23.142ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47372025-04-30T06:38:23.142ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
47382025-04-30T06:38:23.143ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47392025-04-30T06:38:23.145ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47402025-04-30T06:38:23.149ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47412025-04-30T06:38:23.149ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47422025-04-30T06:38:23.149ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47432025-04-30T06:38:23.149ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47442025-04-30T06:38:23.180ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
47452025-04-30T06:38:23.180ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
47462025-04-30T06:38:23.182ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 34 b7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 4............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60610 req_id = 3ec7f8fe-c31f-44cf-9ff6-476668b060ec
47472025-04-30T06:38:23.182ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13495,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60610 req_id = 3ec7f8fe-c31f-44cf-9ff6-476668b060ec
47482025-04-30T06:38:23.182ZDEBGtest_quotas (store): query key key = _mgs._tcp
47492025-04-30T06:38:23.182ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47502025-04-30T06:38:23.182ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60610 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3ec7f8fe-c31f-44cf-9ff6-476668b060ec
47512025-04-30T06:38:23.183ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3967, tv_nsec: 982283136 } })
47522025-04-30T06:38:23.183ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
47532025-04-30T06:38:23.184ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ae e1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64669 req_id = 3ec7c0f9-dfbe-4208-ac06-b9fd20eac5a2
47542025-04-30T06:38:23.184ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44769,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64669 req_id = 3ec7c0f9-dfbe-4208-ac06-b9fd20eac5a2
47552025-04-30T06:38:23.184ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47562025-04-30T06:38:23.184ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64669 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3ec7c0f9-dfbe-4208-ac06-b9fd20eac5a2
47572025-04-30T06:38:23.241ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
47582025-04-30T06:38:23.242ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
47592025-04-30T06:38:23.242ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: eb e0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63501 req_id = 49467661-96d1-458d-ae03-d6126298e185
47602025-04-30T06:38:23.242ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60384,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63501 req_id = 49467661-96d1-458d-ae03-d6126298e185
47612025-04-30T06:38:23.242ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
47622025-04-30T06:38:23.242ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60384, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63501 req_id = 49467661-96d1-458d-ae03-d6126298e185
47632025-04-30T06:38:23.242ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
47642025-04-30T06:38:23.243ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
47652025-04-30T06:38:23.243ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
47662025-04-30T06:38:23.245ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d9c2fc5-773d-437c-8b15-b1bd7e93ad3b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47672025-04-30T06:38:23.245ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
47682025-04-30T06:38:23.248ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8e18004-bf21-4a8d-9d64-f2ee92479e4e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47692025-04-30T06:38:23.248ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
47702025-04-30T06:38:23.271ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a578830b-7c33-4c2f-9061-9fa37210db67", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47712025-04-30T06:38:23.271ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
47722025-04-30T06:38:23.280ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29408efd-de4d-4dc9-9465-4084b67ed49d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47732025-04-30T06:38:23.280ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
47742025-04-30T06:38:23.284ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
47752025-04-30T06:38:23.288ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8abcc3ad-6705-4d9d-b6a6-0f0fdc84aeef", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47762025-04-30T06:38:23.288ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
47772025-04-30T06:38:23.290ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33014eb6-71af-45b9-974e-0575e9f347af", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47782025-04-30T06:38:23.291ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
47792025-04-30T06:38:23.293ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5f1279a-c911-4f83-bed3-5243ac8fa88d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47802025-04-30T06:38:23.293ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
47812025-04-30T06:38:23.298ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df9b388b-26e3-4450-8b68-23dc07080ffd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47822025-04-30T06:38:23.299ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
47832025-04-30T06:38:23.299ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8a0465c-51d9-4734-ad60-04dd660b152a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47842025-04-30T06:38:23.300ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
47852025-04-30T06:38:23.301ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dd06f59-84c9-4802-888d-1bdcd87eba69", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47862025-04-30T06:38:23.301ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
47872025-04-30T06:38:23.302ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcfdc89a-e3af-4c71-9e0b-e99f12bed7af", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47882025-04-30T06:38:23.302ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
47892025-04-30T06:38:23.303ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a46309b3-370a-42fe-9cda-a7dafef70058", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47902025-04-30T06:38:23.303ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
47912025-04-30T06:38:23.304ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a99c067-3622-4160-a61d-243a78f6a032", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47922025-04-30T06:38:23.304ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
47932025-04-30T06:38:23.306ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e25fde5e-2f75-4dad-a38f-e92e8956a263", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47942025-04-30T06:38:23.306ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
47952025-04-30T06:38:23.308ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07efcf67-0cca-472c-b7f0-8b57089d1f56", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47962025-04-30T06:38:23.308ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
47972025-04-30T06:38:23.310ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ee2ff78-53ee-4248-affa-407fa5b9a543", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
47982025-04-30T06:38:23.310ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
47992025-04-30T06:38:23.313ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2df60895-d671-48b4-8b89-57b32706e0c6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48002025-04-30T06:38:23.313ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
48012025-04-30T06:38:23.315ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67c9b910-acd7-4199-bfff-f34565d0462b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48022025-04-30T06:38:23.315ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
48032025-04-30T06:38:23.317ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed0053f8-b377-4ce5-bfdb-6ed1d6f99639", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48042025-04-30T06:38:23.317ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
48052025-04-30T06:38:23.318ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e0bce53-fb97-4821-a6d3-3ce7fde69a4f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48062025-04-30T06:38:23.318ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
48072025-04-30T06:38:23.319ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7034c29-31cc-4184-bbc6-9d9aee392933", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48082025-04-30T06:38:23.319ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
48092025-04-30T06:38:23.321ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "645f0318-a8aa-420f-a685-eeacefb50f0c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48102025-04-30T06:38:23.322ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
48112025-04-30T06:38:23.327ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57992df6-5074-4da0-86cf-f0dab7c75877", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48122025-04-30T06:38:23.327ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
48132025-04-30T06:38:23.330ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51c50b15-0968-44ba-b6a9-bb424a828ed0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48142025-04-30T06:38:23.330ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
48152025-04-30T06:38:23.333ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37f11215-2190-42c3-856e-1a78ecb3da64", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48162025-04-30T06:38:23.333ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
48172025-04-30T06:38:23.335ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88014d43-ec40-4223-a3b6-3dbec0ca2002", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48182025-04-30T06:38:23.335ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
48192025-04-30T06:38:23.336ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48202025-04-30T06:38:23.336ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48212025-04-30T06:38:23.336ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48222025-04-30T06:38:23.336ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
48232025-04-30T06:38:23.337ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48242025-04-30T06:38:23.338ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d31765c-df25-4ac9-bf0f-0125639e3bd7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48252025-04-30T06:38:23.338ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
48262025-04-30T06:38:23.340ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48272025-04-30T06:38:23.350ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48282025-04-30T06:38:23.350ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48292025-04-30T06:38:23.350ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
48302025-04-30T06:38:23.351ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48312025-04-30T06:38:23.354ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48322025-04-30T06:38:23.354ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
48332025-04-30T06:38:23.355ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48342025-04-30T06:38:23.355ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25217789-c7b4-4af3-91d6-75f29e509e8f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48352025-04-30T06:38:23.356ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
48362025-04-30T06:38:23.358ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "408fae1e-f46c-42e8-a0bd-f980d9d5a390", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48372025-04-30T06:38:23.358ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
48382025-04-30T06:38:23.363ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "014d3d63-3097-4ef6-b1d5-ec390cb6c031", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48392025-04-30T06:38:23.363ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
48402025-04-30T06:38:23.364ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebbe31d5-ee61-45f3-a467-d6fa4e3f47d4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48412025-04-30T06:38:23.365ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
48422025-04-30T06:38:23.368ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1754a9e4-0792-49b3-8cb6-8d6a0d8abe60", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48432025-04-30T06:38:23.368ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
48442025-04-30T06:38:23.370ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f78e51c0-81cc-47bf-87f8-fc8133a35adc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48452025-04-30T06:38:23.370ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
48462025-04-30T06:38:23.373ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c15ad15-69c5-4878-a088-b15032297f67", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48472025-04-30T06:38:23.373ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
48482025-04-30T06:38:23.377ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61cf9539-1dc7-4c04-ae58-622e1bf1fd9b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48492025-04-30T06:38:23.378ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
48502025-04-30T06:38:23.383ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "708a20cd-9635-43c8-9321-115865651683", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48512025-04-30T06:38:23.384ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
48522025-04-30T06:38:23.387ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
48532025-04-30T06:38:23.389ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20e5ddad-1747-4bd8-bdb8-d4d3f8209b60", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48542025-04-30T06:38:23.389ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
48552025-04-30T06:38:23.391ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72022205-618f-4044-84f8-d919e97f747a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48562025-04-30T06:38:23.391ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
48572025-04-30T06:38:23.401ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "411e7cb1-eaaf-4f0d-8088-d24b06644bf0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48582025-04-30T06:38:23.401ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
48592025-04-30T06:38:23.403ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0de8c13b-1875-4a5b-a340-910f96318d64", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48602025-04-30T06:38:23.403ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
48612025-04-30T06:38:23.405ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "302931a8-d46b-44bd-87ac-b6f713205aba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48622025-04-30T06:38:23.405ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
48632025-04-30T06:38:23.406ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe61fe64-ec70-4372-a1e0-f6aa7794f25b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48642025-04-30T06:38:23.406ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
48652025-04-30T06:38:23.407ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b3cf0a3-70ee-473c-a620-e88ca95d3f65", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48662025-04-30T06:38:23.407ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
48672025-04-30T06:38:23.408ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b56fe529-a7ff-4c9d-9752-1cf380f85a36", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48682025-04-30T06:38:23.408ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
48692025-04-30T06:38:23.409ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "850d2665-f991-40ad-8baf-5d5d116123ff", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
48702025-04-30T06:38:23.409ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48712025-04-30T06:38:23.410ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48722025-04-30T06:38:23.410ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48732025-04-30T06:38:23.410ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48742025-04-30T06:38:23.410ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48752025-04-30T06:38:23.414ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48762025-04-30T06:38:23.420ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48772025-04-30T06:38:23.420ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48782025-04-30T06:38:23.420ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48792025-04-30T06:38:23.420ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48802025-04-30T06:38:23.420ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48812025-04-30T06:38:23.421ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48822025-04-30T06:38:23.426ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48832025-04-30T06:38:23.432ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48842025-04-30T06:38:23.433ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48852025-04-30T06:38:23.433ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48862025-04-30T06:38:23.435ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48872025-04-30T06:38:23.438ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48882025-04-30T06:38:23.438ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48892025-04-30T06:38:23.438ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48902025-04-30T06:38:23.438ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
48912025-04-30T06:38:23.439ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48922025-04-30T06:38:23.442ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48932025-04-30T06:38:23.450ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48942025-04-30T06:38:23.450ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48952025-04-30T06:38:23.450ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
48962025-04-30T06:38:23.451ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48972025-04-30T06:38:23.526ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
48982025-04-30T06:38:23.526ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
48992025-04-30T06:38:23.527ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:54155 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
49002025-04-30T06:38:23.528ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:54155 req_id = cfaa9b89-526f-478b-b57c-288ac07103b5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49012025-04-30T06:38:23.528ZINFOtest_quotas (dropshot): request completed latency_us = 462 local_addr = [::1]:55655 method = GET remote_addr = [::1]:54155 req_id = cfaa9b89-526f-478b-b57c-288ac07103b5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49022025-04-30T06:38:23.528ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49032025-04-30T06:38:23.528ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
49042025-04-30T06:38:23.530ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49052025-04-30T06:38:23.531ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfaa9b89-526f-478b-b57c-288ac07103b5", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49062025-04-30T06:38:23.534ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49072025-04-30T06:38:23.534ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49082025-04-30T06:38:23.534ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
49092025-04-30T06:38:23.537ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49102025-04-30T06:38:23.537ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49112025-04-30T06:38:23.537ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49122025-04-30T06:38:23.537ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49132025-04-30T06:38:23.538ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49142025-04-30T06:38:23.543ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49152025-04-30T06:38:23.548ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49162025-04-30T06:38:23.548ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49172025-04-30T06:38:23.548ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49182025-04-30T06:38:23.549ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49192025-04-30T06:38:23.586ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
49202025-04-30T06:38:23.591ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
49212025-04-30T06:38:23.591ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
49222025-04-30T06:38:23.591ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 450.961802ms iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
49232025-04-30T06:38:23.591ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
49242025-04-30T06:38:23.591ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49252025-04-30T06:38:23.592ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49262025-04-30T06:38:23.592ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49272025-04-30T06:38:23.592ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49282025-04-30T06:38:23.594ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49292025-04-30T06:38:23.597ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49302025-04-30T06:38:23.602ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49312025-04-30T06:38:23.602ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49322025-04-30T06:38:23.602ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49332025-04-30T06:38:23.602ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49342025-04-30T06:38:23.606ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
49352025-04-30T06:38:23.606ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
49362025-04-30T06:38:23.607ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 52 31 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 R1..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57079 req_id = 30a845b1-7dd1-446a-8390-9236745c680c
49372025-04-30T06:38:23.607ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21041,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57079 req_id = 30a845b1-7dd1-446a-8390-9236745c680c
49382025-04-30T06:38:23.607ZDEBGtest_quotas (store): query key key = _mgs._tcp
49392025-04-30T06:38:23.607ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49402025-04-30T06:38:23.607ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57079 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 30a845b1-7dd1-446a-8390-9236745c680c
49412025-04-30T06:38:23.608ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3968, tv_nsec: 406986220 } })
49422025-04-30T06:38:23.608ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
49432025-04-30T06:38:23.609ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 64 f8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e d...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52194 req_id = 65b5dca0-c1ba-41d2-a1f6-eba8319238fa
49442025-04-30T06:38:23.609ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25848,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52194 req_id = 65b5dca0-c1ba-41d2-a1f6-eba8319238fa
49452025-04-30T06:38:23.609ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49462025-04-30T06:38:23.610ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52194 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 65b5dca0-c1ba-41d2-a1f6-eba8319238fa
49472025-04-30T06:38:23.675ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
49482025-04-30T06:38:23.675ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49492025-04-30T06:38:23.676ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49502025-04-30T06:38:23.676ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49512025-04-30T06:38:23.676ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
49522025-04-30T06:38:23.678ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49532025-04-30T06:38:23.678ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 98 1d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54314 req_id = ebdb4118-2835-4a59-a4fc-17d0a0eda686
49542025-04-30T06:38:23.679ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38941,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54314 req_id = ebdb4118-2835-4a59-a4fc-17d0a0eda686
49552025-04-30T06:38:23.679ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
49562025-04-30T06:38:23.679ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38941, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54314 req_id = ebdb4118-2835-4a59-a4fc-17d0a0eda686
49572025-04-30T06:38:23.680ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
49582025-04-30T06:38:23.680ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
49592025-04-30T06:38:23.680ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
49602025-04-30T06:38:23.681ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49612025-04-30T06:38:23.688ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49622025-04-30T06:38:23.688ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49632025-04-30T06:38:23.688ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
49642025-04-30T06:38:23.689ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49652025-04-30T06:38:23.690ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c045037b-8cdb-4f1d-a22b-016df4802f25", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49662025-04-30T06:38:23.691ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
49672025-04-30T06:38:23.697ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49682025-04-30T06:38:23.698ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
49692025-04-30T06:38:23.698ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49702025-04-30T06:38:23.701ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "694af442-6d7c-4e82-9e6c-a75e3e1188f9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49712025-04-30T06:38:23.701ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
49722025-04-30T06:38:23.704ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66b31232-c175-4e28-a112-ce76b658b7d6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49732025-04-30T06:38:23.704ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
49742025-04-30T06:38:23.706ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd0ce3ff-1049-44fb-8b54-53555fc546c5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49752025-04-30T06:38:23.706ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
49762025-04-30T06:38:23.710ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "abb0e119-5b06-478a-b6a5-db3f499dc5fb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49772025-04-30T06:38:23.710ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
49782025-04-30T06:38:23.714ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d25844d-e4ff-4c7a-aea8-220937c181f3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49792025-04-30T06:38:23.714ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
49802025-04-30T06:38:23.719ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a1127b6-615c-4037-a760-8c31706c1f06", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49812025-04-30T06:38:23.719ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
49822025-04-30T06:38:23.730ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71679fb3-8418-4825-96c0-62a0d1ff1cbf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49832025-04-30T06:38:23.731ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
49842025-04-30T06:38:23.733ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34b7f9be-672d-46a2-9a07-ab724491c672", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49852025-04-30T06:38:23.734ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
49862025-04-30T06:38:23.737ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f74b00b-aa8a-4dda-b321-86bcdfd38716", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49872025-04-30T06:38:23.737ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
49882025-04-30T06:38:23.738ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
49892025-04-30T06:38:23.739ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
49902025-04-30T06:38:23.739ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01c1b083-fc72-4c7c-9e55-2e4e4330d932", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49912025-04-30T06:38:23.739ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
49922025-04-30T06:38:23.740ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49932025-04-30T06:38:23.741ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = bd0cae91-07db-4f6c-95f2-6af18b50d4f3 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49942025-04-30T06:38:23.741ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = bd0cae91-07db-4f6c-95f2-6af18b50d4f3 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49952025-04-30T06:38:23.741ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = bd0cae91-07db-4f6c-95f2-6af18b50d4f3 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49962025-04-30T06:38:23.742ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = bd0cae91-07db-4f6c-95f2-6af18b50d4f3 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
49972025-04-30T06:38:23.743ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3fd8d3c-f718-4fff-80b6-7b21cc3f1f67", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
49982025-04-30T06:38:23.743ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
49992025-04-30T06:38:23.745ZINFOtest_quotas (dropshot_internal): request completed latency_us = 4844 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = bd0cae91-07db-4f6c-95f2-6af18b50d4f3 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
50002025-04-30T06:38:23.746ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd0cae91-07db-4f6c-95f2-6af18b50d4f3", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50012025-04-30T06:38:23.746ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50022025-04-30T06:38:23.746ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c34b5b3-d0b3-470c-beed-0497e41f55e7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50032025-04-30T06:38:23.746ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
50042025-04-30T06:38:23.746ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 52338d96-71a9-477a-9d80-c6bacc2e235a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50052025-04-30T06:38:23.747ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 52338d96-71a9-477a-9d80-c6bacc2e235a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50062025-04-30T06:38:23.747ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 52338d96-71a9-477a-9d80-c6bacc2e235a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50072025-04-30T06:38:23.748ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 52338d96-71a9-477a-9d80-c6bacc2e235a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50082025-04-30T06:38:23.751ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8bf40f3-fec6-4121-8624-9da35fa2d831", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50092025-04-30T06:38:23.751ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
50102025-04-30T06:38:23.752ZINFOtest_quotas (dropshot_internal): request completed latency_us = 5098 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 52338d96-71a9-477a-9d80-c6bacc2e235a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
50112025-04-30T06:38:23.752ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "52338d96-71a9-477a-9d80-c6bacc2e235a", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50122025-04-30T06:38:23.752ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
50132025-04-30T06:38:23.752ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50142025-04-30T06:38:23.752ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
50152025-04-30T06:38:23.756ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a73654b-0691-4441-ab5e-2bca1285269e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50162025-04-30T06:38:23.756ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
50172025-04-30T06:38:23.758ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "783654d9-6b76-4bcf-8b39-1fb4e645b51c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50182025-04-30T06:38:23.758ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
50192025-04-30T06:38:23.761ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d2f2423-bc81-4a9b-bb6e-0cea36fe4271", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50202025-04-30T06:38:23.761ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
50212025-04-30T06:38:23.765ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f63f1832-55b3-4271-87fe-f96ae650b8a9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50222025-04-30T06:38:23.769ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
50232025-04-30T06:38:23.770ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d94331f5-d89c-4b8d-ab4b-213df9f8ebdf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50242025-04-30T06:38:23.770ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
50252025-04-30T06:38:23.771ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e825de15-c886-4bd0-a755-56d10a15189b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50262025-04-30T06:38:23.771ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
50272025-04-30T06:38:23.773ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a36a51d5-b419-4bc7-ba4c-7d62d6ba7b1f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50282025-04-30T06:38:23.773ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
50292025-04-30T06:38:23.775ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "030991fe-6aae-4854-9ad6-02cf794a685e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50302025-04-30T06:38:23.776ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
50312025-04-30T06:38:23.777ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee8b0fb9-892b-4f56-afda-92a7f4edcf9e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50322025-04-30T06:38:23.778ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
50332025-04-30T06:38:23.780ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50b25b71-f7bd-4c49-888d-a7ad77caafe4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50342025-04-30T06:38:23.781ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
50352025-04-30T06:38:23.783ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3933fb92-9dab-4675-b2e0-cafa88b89414", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50362025-04-30T06:38:23.783ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
50372025-04-30T06:38:23.785ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99ee1eb4-4ba9-4296-a00e-dd6289e9474b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50382025-04-30T06:38:23.786ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
50392025-04-30T06:38:23.788ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fb0da4a-34cd-4758-b2f0-91a024dfd53c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50402025-04-30T06:38:23.788ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
50412025-04-30T06:38:23.789ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50422025-04-30T06:38:23.789ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50432025-04-30T06:38:23.789ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50442025-04-30T06:38:23.789ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
50452025-04-30T06:38:23.790ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50462025-04-30T06:38:23.793ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7983734c-8e5d-4e23-a685-1c9f44047f76", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50472025-04-30T06:38:23.793ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
50482025-04-30T06:38:23.793ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50492025-04-30T06:38:23.799ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50502025-04-30T06:38:23.799ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50512025-04-30T06:38:23.800ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
50522025-04-30T06:38:23.800ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50532025-04-30T06:38:23.802ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6144d6ff-8adb-419f-a71c-1cd53b268d76", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50542025-04-30T06:38:23.803ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
50552025-04-30T06:38:23.803ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50562025-04-30T06:38:23.803ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
50572025-04-30T06:38:23.804ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50582025-04-30T06:38:23.806ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8740331e-7248-4d29-b55d-7e7397969ea5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50592025-04-30T06:38:23.807ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
50602025-04-30T06:38:23.811ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c798853-250d-43aa-be2a-03f1e2ebcab6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50612025-04-30T06:38:23.812ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
50622025-04-30T06:38:23.813ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9ddc9ae-28cb-4976-a623-986826134e2d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50632025-04-30T06:38:23.814ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
50642025-04-30T06:38:23.814ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86c5e23d-501c-4683-885a-9e717d4e714a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50652025-04-30T06:38:23.814ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
50662025-04-30T06:38:23.815ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7964a155-e92a-424a-88c1-e2271863eb7e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50672025-04-30T06:38:23.815ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
50682025-04-30T06:38:23.817ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9e857f3-4059-4418-9e40-9a21879ba91e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50692025-04-30T06:38:23.817ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
50702025-04-30T06:38:23.819ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebbb5d88-efc4-44ae-b965-9bdc9532bd9b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50712025-04-30T06:38:23.820ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
50722025-04-30T06:38:23.821ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "969e1bf7-2def-4919-bc06-2dae3161d515", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50732025-04-30T06:38:23.822ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
50742025-04-30T06:38:23.824ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2672e0d4-a824-4336-b0bd-7082580378cb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50752025-04-30T06:38:23.824ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
50762025-04-30T06:38:23.826ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cdd0490-74f6-4acd-bdff-52ee93d206e3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50772025-04-30T06:38:23.826ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
50782025-04-30T06:38:23.829ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f784ef68-6d47-4f22-a82f-2999ccbe4d9e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50792025-04-30T06:38:23.829ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
50802025-04-30T06:38:23.833ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78176a02-ee1f-465d-895a-9268b87926ba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50812025-04-30T06:38:23.833ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
50822025-04-30T06:38:23.834ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
50832025-04-30T06:38:23.836ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94e4a526-702a-418e-96ff-2e31f0c601e9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50842025-04-30T06:38:23.837ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
50852025-04-30T06:38:23.838ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0ca6026-9bc1-4c85-b1c0-51cf56a809fc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50862025-04-30T06:38:23.838ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
50872025-04-30T06:38:23.840ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d81a61a-3f36-4983-853e-02a5e8d73d6d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50882025-04-30T06:38:23.840ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
50892025-04-30T06:38:23.842ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "410d53c2-d3ac-4f9e-9eb7-1eecf4a74c86", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
50902025-04-30T06:38:23.842ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50912025-04-30T06:38:23.842ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50922025-04-30T06:38:23.842ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50932025-04-30T06:38:23.842ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50942025-04-30T06:38:23.842ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50952025-04-30T06:38:23.845ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50962025-04-30T06:38:23.851ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50972025-04-30T06:38:23.851ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50982025-04-30T06:38:23.851ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50992025-04-30T06:38:23.851ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51002025-04-30T06:38:23.851ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51012025-04-30T06:38:23.851ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51022025-04-30T06:38:23.851ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
51032025-04-30T06:38:23.852ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
51042025-04-30T06:38:23.855ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51052025-04-30T06:38:23.859ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
51062025-04-30T06:38:23.859ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51072025-04-30T06:38:23.859ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51082025-04-30T06:38:23.860ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51092025-04-30T06:38:23.860ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
51102025-04-30T06:38:23.861ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
51112025-04-30T06:38:23.861ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
51122025-04-30T06:38:23.862ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
51132025-04-30T06:38:23.908ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
51142025-04-30T06:38:23.908ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
51152025-04-30T06:38:23.909ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51162025-04-30T06:38:23.909ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51172025-04-30T06:38:23.909ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51182025-04-30T06:38:23.909ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
51192025-04-30T06:38:23.910ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51202025-04-30T06:38:23.910ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:65492 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
51212025-04-30T06:38:23.911ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:65492 req_id = 5673f8b3-b13a-42bb-8f04-4048525a9c82 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51222025-04-30T06:38:23.912ZINFOtest_quotas (dropshot): request completed latency_us = 563 local_addr = [::1]:55655 method = GET remote_addr = [::1]:65492 req_id = 5673f8b3-b13a-42bb-8f04-4048525a9c82 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51232025-04-30T06:38:23.913ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5673f8b3-b13a-42bb-8f04-4048525a9c82", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
51242025-04-30T06:38:23.917ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51252025-04-30T06:38:23.917ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51262025-04-30T06:38:23.917ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
51272025-04-30T06:38:23.919ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51282025-04-30T06:38:23.919ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51292025-04-30T06:38:23.919ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51302025-04-30T06:38:23.919ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51312025-04-30T06:38:23.919ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51322025-04-30T06:38:23.921ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51332025-04-30T06:38:23.925ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51342025-04-30T06:38:23.926ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51352025-04-30T06:38:23.926ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
51362025-04-30T06:38:23.926ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51372025-04-30T06:38:23.928ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51382025-04-30T06:38:23.933ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51392025-04-30T06:38:23.933ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51402025-04-30T06:38:23.933ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51412025-04-30T06:38:23.934ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51422025-04-30T06:38:23.934ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51432025-04-30T06:38:23.934ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
51442025-04-30T06:38:23.935ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51452025-04-30T06:38:23.987ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
51462025-04-30T06:38:23.987ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
51472025-04-30T06:38:23.987ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 396.026266ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
51482025-04-30T06:38:23.987ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
51492025-04-30T06:38:23.987ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51502025-04-30T06:38:23.988ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51512025-04-30T06:38:23.988ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51522025-04-30T06:38:23.988ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51532025-04-30T06:38:23.989ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51542025-04-30T06:38:23.990ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
51552025-04-30T06:38:24.001ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51562025-04-30T06:38:24.007ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51572025-04-30T06:38:24.008ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51582025-04-30T06:38:24.008ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51592025-04-30T06:38:24.008ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51602025-04-30T06:38:24.016ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
51612025-04-30T06:38:24.016ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
51622025-04-30T06:38:24.017ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8c ff 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50201 req_id = b2004bba-0f06-478c-9e7b-7aa00b480c33
51632025-04-30T06:38:24.018ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36095,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50201 req_id = b2004bba-0f06-478c-9e7b-7aa00b480c33
51642025-04-30T06:38:24.018ZDEBGtest_quotas (store): query key key = _mgs._tcp
51652025-04-30T06:38:24.018ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51662025-04-30T06:38:24.018ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50201 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b2004bba-0f06-478c-9e7b-7aa00b480c33
51672025-04-30T06:38:24.019ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3968, tv_nsec: 818180968 } })
51682025-04-30T06:38:24.019ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
51692025-04-30T06:38:24.019ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 56 d4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e V...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54236 req_id = 413675f5-698e-4d58-b928-e219fba442bc
51702025-04-30T06:38:24.019ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22228,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54236 req_id = 413675f5-698e-4d58-b928-e219fba442bc
51712025-04-30T06:38:24.019ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51722025-04-30T06:38:24.020ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54236 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 413675f5-698e-4d58-b928-e219fba442bc
51732025-04-30T06:38:24.079ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
51742025-04-30T06:38:24.090ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51752025-04-30T06:38:24.090ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51762025-04-30T06:38:24.090ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51772025-04-30T06:38:24.090ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
51782025-04-30T06:38:24.090ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51792025-04-30T06:38:24.090ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7c 46 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c |F..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65022 req_id = 0aa676e9-f478-4903-a131-9cbd63c1af9c
51802025-04-30T06:38:24.090ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31814,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65022 req_id = 0aa676e9-f478-4903-a131-9cbd63c1af9c
51812025-04-30T06:38:24.090ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
51822025-04-30T06:38:24.090ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31814, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65022 req_id = 0aa676e9-f478-4903-a131-9cbd63c1af9c
51832025-04-30T06:38:24.090ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
51842025-04-30T06:38:24.090ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
51852025-04-30T06:38:24.090ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
51862025-04-30T06:38:24.090ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c014fc7b-3942-49ec-a1e5-8af8646b2416", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
51872025-04-30T06:38:24.090ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
51882025-04-30T06:38:24.090ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51892025-04-30T06:38:24.092ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51902025-04-30T06:38:24.092ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51912025-04-30T06:38:24.092ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
51922025-04-30T06:38:24.093ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51932025-04-30T06:38:24.095ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "085d4e84-f558-4a25-a75b-a6fc3be00eb4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
51942025-04-30T06:38:24.095ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
51952025-04-30T06:38:24.096ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51962025-04-30T06:38:24.096ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
51972025-04-30T06:38:24.098ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51982025-04-30T06:38:24.104ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
51992025-04-30T06:38:24.105ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52002025-04-30T06:38:24.105ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
52012025-04-30T06:38:24.106ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
52022025-04-30T06:38:24.114ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70ee6ba0-bcfd-43f1-a87d-c0b382853f40", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52032025-04-30T06:38:24.114ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
52042025-04-30T06:38:24.115ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
52052025-04-30T06:38:24.115ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
52062025-04-30T06:38:24.123ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
52072025-04-30T06:38:24.123ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52082025-04-30T06:38:24.124ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34a1e0bc-1889-4ea0-95a8-0d62f84faabc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52092025-04-30T06:38:24.124ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
52102025-04-30T06:38:24.127ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d7c6f51-6f08-4e69-8e8f-2e23b24b0f61", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52112025-04-30T06:38:24.127ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
52122025-04-30T06:38:24.130ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "643ec333-14a1-412d-84c8-c560d337b4f3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52132025-04-30T06:38:24.131ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
52142025-04-30T06:38:24.133ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "faf789d0-86ef-4d84-8b45-0469d344a226", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52152025-04-30T06:38:24.133ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
52162025-04-30T06:38:24.135ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22b24749-da04-4563-a1b1-76c976a87f8c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52172025-04-30T06:38:24.135ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
52182025-04-30T06:38:24.136ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f364c43a-97fa-47f6-8055-5b0b80bd21c8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52192025-04-30T06:38:24.137ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
52202025-04-30T06:38:24.138ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5165bb5-394e-49e3-a7b7-e1bcec040a1c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52212025-04-30T06:38:24.138ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
52222025-04-30T06:38:24.139ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b6e5ad0-bc3d-492f-98f7-71843f447add", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52232025-04-30T06:38:24.139ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
52242025-04-30T06:38:24.143ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3256c066-13da-4bc0-bb3b-66dd3fd619d8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52252025-04-30T06:38:24.143ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
52262025-04-30T06:38:24.143ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
52272025-04-30T06:38:24.145ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bb04483-499e-4a3a-947c-224c1349a27b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52282025-04-30T06:38:24.145ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
52292025-04-30T06:38:24.146ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7232a875-0e41-4f1c-b652-fd907e530410", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52302025-04-30T06:38:24.147ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
52312025-04-30T06:38:24.148ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bf7183d-c148-4d86-8bba-5f32faa8a950", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52322025-04-30T06:38:24.149ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
52332025-04-30T06:38:24.150ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "441dabed-8cf6-4a00-a238-f3b551da4769", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52342025-04-30T06:38:24.150ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
52352025-04-30T06:38:24.154ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce064966-eedb-404d-ace6-5b9a3cd6ec06", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52362025-04-30T06:38:24.154ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
52372025-04-30T06:38:24.156ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4f6322d-977c-4d10-8e22-1fc9c3138f1d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52382025-04-30T06:38:24.156ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
52392025-04-30T06:38:24.160ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a82b70a0-b385-4318-b30c-106be6c987b5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52402025-04-30T06:38:24.163ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
52412025-04-30T06:38:24.164ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f245ba0c-6fc6-4cb0-986c-3a40ea1a5347", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52422025-04-30T06:38:24.164ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
52432025-04-30T06:38:24.165ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79fd943b-91cf-4401-952c-02542c02217f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52442025-04-30T06:38:24.165ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
52452025-04-30T06:38:24.167ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6890be55-0080-49c8-96e7-678e1a03083a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52462025-04-30T06:38:24.167ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
52472025-04-30T06:38:24.168ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5294d010-015b-4e96-8f33-bb2e6553572c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52482025-04-30T06:38:24.168ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
52492025-04-30T06:38:24.170ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e49fc907-a7c1-42a9-9905-99987f686c6b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52502025-04-30T06:38:24.182ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
52512025-04-30T06:38:24.186ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf883eb1-5f23-4936-9e86-540b07773870", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52522025-04-30T06:38:24.186ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
52532025-04-30T06:38:24.188ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b2906f0-58fe-4229-84dd-69b894d5149c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52542025-04-30T06:38:24.188ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
52552025-04-30T06:38:24.190ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d22bb2b-d6d0-4c5e-9a73-7133ba02ca5c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52562025-04-30T06:38:24.190ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
52572025-04-30T06:38:24.191ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7dfbb4f-deee-403d-b353-0f4929fc9de5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:23 GMT"} })
52582025-04-30T06:38:24.192ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
52592025-04-30T06:38:24.194ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52602025-04-30T06:38:24.195ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52612025-04-30T06:38:24.195ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52622025-04-30T06:38:24.195ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
52632025-04-30T06:38:24.196ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52642025-04-30T06:38:24.215ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52652025-04-30T06:38:24.245ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52662025-04-30T06:38:24.246ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52672025-04-30T06:38:24.246ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
52682025-04-30T06:38:24.246ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52692025-04-30T06:38:24.247ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
52702025-04-30T06:38:24.259ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9847578e-d5d8-4607-92e8-3a7cc1b0099a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52712025-04-30T06:38:24.260ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
52722025-04-30T06:38:24.260ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52732025-04-30T06:38:24.261ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
52742025-04-30T06:38:24.261ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52752025-04-30T06:38:24.264ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c81a925b-61df-418b-89e5-ca300427df97", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52762025-04-30T06:38:24.264ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
52772025-04-30T06:38:24.265ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb0af61d-be90-48eb-8860-9d0b60a357f1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52782025-04-30T06:38:24.265ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
52792025-04-30T06:38:24.266ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d857faf-51cd-4f2f-b647-146bf85a0755", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52802025-04-30T06:38:24.266ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
52812025-04-30T06:38:24.268ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77d80364-6886-4051-8aca-5b249400b84b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52822025-04-30T06:38:24.268ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
52832025-04-30T06:38:24.269ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0c7bdcc-8fc8-4ed6-978e-c3e787839beb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52842025-04-30T06:38:24.269ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
52852025-04-30T06:38:24.271ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8461ef31-7b98-45fb-92b7-3d7953c5b510", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52862025-04-30T06:38:24.272ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
52872025-04-30T06:38:24.307ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cebdd361-be24-49ff-a2f7-068f302efe7b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52882025-04-30T06:38:24.307ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
52892025-04-30T06:38:24.313ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d9775ec-dbc4-4c86-a6f9-4412a6121c53", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52902025-04-30T06:38:24.313ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
52912025-04-30T06:38:24.315ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51c02167-a6c7-46aa-ba13-752cd591cca3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52922025-04-30T06:38:24.315ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
52932025-04-30T06:38:24.318ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92bba18b-56c9-4e92-bd79-875cfb6b07ea", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52942025-04-30T06:38:24.318ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
52952025-04-30T06:38:24.320ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d82dcd8-3e41-4828-a794-928fb14d7f0a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52962025-04-30T06:38:24.320ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
52972025-04-30T06:38:24.323ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ca878c7-919b-43ea-a43d-7535ada64064", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
52982025-04-30T06:38:24.323ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
52992025-04-30T06:38:24.326ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e71a414-e6c4-4dbb-96ad-1ef386714b91", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
53002025-04-30T06:38:24.326ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
53012025-04-30T06:38:24.326ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
53022025-04-30T06:38:24.327ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0c90902-9d85-4b07-b5f3-c419c25584bb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
53032025-04-30T06:38:24.327ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
53042025-04-30T06:38:24.328ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af285100-69b2-46f3-9432-1eabf716eee1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
53052025-04-30T06:38:24.328ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
53062025-04-30T06:38:24.329ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ffcb5b8-a870-4575-93ea-787c0be546dd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
53072025-04-30T06:38:24.329ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53082025-04-30T06:38:24.329ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53092025-04-30T06:38:24.329ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53102025-04-30T06:38:24.329ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53112025-04-30T06:38:24.330ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53122025-04-30T06:38:24.371ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53132025-04-30T06:38:24.378ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53142025-04-30T06:38:24.378ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53152025-04-30T06:38:24.378ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53162025-04-30T06:38:24.378ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53172025-04-30T06:38:24.378ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53182025-04-30T06:38:24.379ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53192025-04-30T06:38:24.380ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53202025-04-30T06:38:24.380ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53212025-04-30T06:38:24.380ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53222025-04-30T06:38:24.380ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
53232025-04-30T06:38:24.380ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53242025-04-30T06:38:24.391ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53252025-04-30T06:38:24.413ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53262025-04-30T06:38:24.413ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53272025-04-30T06:38:24.413ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53282025-04-30T06:38:24.413ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53292025-04-30T06:38:24.416ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53302025-04-30T06:38:24.452ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53312025-04-30T06:38:24.452ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53322025-04-30T06:38:24.452ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
53332025-04-30T06:38:24.452ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53342025-04-30T06:38:24.623ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
53352025-04-30T06:38:24.623ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
53362025-04-30T06:38:24.629ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:43922 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
53372025-04-30T06:38:24.629ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:43922 req_id = 6964c7e6-ab17-48ab-873d-37fc0cc968db sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53382025-04-30T06:38:24.630ZINFOtest_quotas (dropshot): request completed latency_us = 645 local_addr = [::1]:55655 method = GET remote_addr = [::1]:43922 req_id = 6964c7e6-ab17-48ab-873d-37fc0cc968db response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53392025-04-30T06:38:24.630ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53402025-04-30T06:38:24.630ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
53412025-04-30T06:38:24.631ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53422025-04-30T06:38:24.631ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6964c7e6-ab17-48ab-873d-37fc0cc968db", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
53432025-04-30T06:38:24.634ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53442025-04-30T06:38:24.634ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53452025-04-30T06:38:24.634ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
53462025-04-30T06:38:24.636ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53472025-04-30T06:38:24.636ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53482025-04-30T06:38:24.636ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53492025-04-30T06:38:24.636ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53502025-04-30T06:38:24.636ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53512025-04-30T06:38:24.640ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53522025-04-30T06:38:24.668ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53532025-04-30T06:38:24.668ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53542025-04-30T06:38:24.668ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53552025-04-30T06:38:24.669ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53562025-04-30T06:38:24.761ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
53572025-04-30T06:38:24.762ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
53582025-04-30T06:38:24.764ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
53592025-04-30T06:38:24.764ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 776.530226ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
53602025-04-30T06:38:24.765ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
53612025-04-30T06:38:24.765ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53622025-04-30T06:38:24.765ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53632025-04-30T06:38:24.765ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53642025-04-30T06:38:24.765ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53652025-04-30T06:38:24.769ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53662025-04-30T06:38:24.774ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53672025-04-30T06:38:24.778ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53682025-04-30T06:38:24.778ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53692025-04-30T06:38:24.778ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53702025-04-30T06:38:24.779ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53712025-04-30T06:38:24.808ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
53722025-04-30T06:38:24.808ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
53732025-04-30T06:38:24.809ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 22 55 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 "U..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57012 req_id = c48ff25b-cc11-4de0-a771-c3d0dc8dd91e
53742025-04-30T06:38:24.810ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8789,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57012 req_id = c48ff25b-cc11-4de0-a771-c3d0dc8dd91e
53752025-04-30T06:38:24.810ZDEBGtest_quotas (store): query key key = _mgs._tcp
53762025-04-30T06:38:24.810ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53772025-04-30T06:38:24.811ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57012 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c48ff25b-cc11-4de0-a771-c3d0dc8dd91e
53782025-04-30T06:38:24.812ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53792025-04-30T06:38:24.812ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53802025-04-30T06:38:24.812ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53812025-04-30T06:38:24.812ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
53822025-04-30T06:38:24.813ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53832025-04-30T06:38:24.813ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3969, tv_nsec: 612693462 } })
53842025-04-30T06:38:24.813ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
53852025-04-30T06:38:24.814ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ac 75 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .u..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65051 req_id = a65ae516-2557-4eb9-a589-dfe351192777
53862025-04-30T06:38:24.814ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44149,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65051 req_id = a65ae516-2557-4eb9-a589-dfe351192777
53872025-04-30T06:38:24.814ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53882025-04-30T06:38:24.814ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65051 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a65ae516-2557-4eb9-a589-dfe351192777
53892025-04-30T06:38:24.853ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
53902025-04-30T06:38:24.854ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53912025-04-30T06:38:24.854ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53922025-04-30T06:38:24.855ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 28 2a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c (*..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60598 req_id = 19b0c0c2-8129-4e57-aabc-e47c07782dda
53932025-04-30T06:38:24.855ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10282,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60598 req_id = 19b0c0c2-8129-4e57-aabc-e47c07782dda
53942025-04-30T06:38:24.855ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
53952025-04-30T06:38:24.855ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10282, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60598 req_id = 19b0c0c2-8129-4e57-aabc-e47c07782dda
53962025-04-30T06:38:24.856ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53972025-04-30T06:38:24.863ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53982025-04-30T06:38:24.863ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53992025-04-30T06:38:24.863ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
54002025-04-30T06:38:24.864ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54012025-04-30T06:38:24.894ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
54022025-04-30T06:38:24.894ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
54032025-04-30T06:38:24.894ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
54042025-04-30T06:38:24.895ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54052025-04-30T06:38:24.896ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54062025-04-30T06:38:24.896ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54072025-04-30T06:38:24.896ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54082025-04-30T06:38:24.910ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54092025-04-30T06:38:24.910ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
54102025-04-30T06:38:24.911ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54112025-04-30T06:38:24.912ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c230e8b3-4836-4154-8c29-31b975bb1acb", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54122025-04-30T06:38:24.912ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
54132025-04-30T06:38:24.914ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e240ac1-814a-44d8-983e-f1f02abe8c9d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54142025-04-30T06:38:24.914ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
54152025-04-30T06:38:24.917ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da595b21-99d6-4721-aa1a-52b281a99c96", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54162025-04-30T06:38:24.917ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
54172025-04-30T06:38:24.920ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "562c8ca1-418a-4ea9-b370-190f18a4bd21", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54182025-04-30T06:38:24.920ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
54192025-04-30T06:38:24.954ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81b909df-621e-4dbd-943a-05c3230c4975", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54202025-04-30T06:38:24.954ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
54212025-04-30T06:38:24.958ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "faa0c358-5fba-4661-92b9-71c23d9b1845", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54222025-04-30T06:38:24.967ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
54232025-04-30T06:38:24.968ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
54242025-04-30T06:38:24.973ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4f9b8d5-fffe-4275-ae66-84361db9fdce", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54252025-04-30T06:38:24.974ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
54262025-04-30T06:38:24.976ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d058a49-9929-4cf1-87c5-6f3b7a8d1d02", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54272025-04-30T06:38:24.976ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
54282025-04-30T06:38:24.977ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da0c41cb-1a0d-4978-8488-d23b9c39a4cb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54292025-04-30T06:38:24.977ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
54302025-04-30T06:38:24.978ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ae898a0-fabc-4f9f-9b1e-887bc4cd5da9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54312025-04-30T06:38:24.978ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
54322025-04-30T06:38:24.979ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "029a84b2-d738-4bff-8139-9321977bd26d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54332025-04-30T06:38:24.979ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
54342025-04-30T06:38:24.985ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9fcc32b-7073-400b-8b40-22d10d19a317", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54352025-04-30T06:38:24.986ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
54362025-04-30T06:38:24.987ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00e47ffd-ba1f-49cd-b18b-da0fc962161c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54372025-04-30T06:38:24.987ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
54382025-04-30T06:38:24.989ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16bf5d92-43ad-43e9-88bb-15f9b9597b21", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54392025-04-30T06:38:24.989ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
54402025-04-30T06:38:24.992ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8d1ddaf-7021-4e83-be99-5ba8e89be521", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54412025-04-30T06:38:24.993ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
54422025-04-30T06:38:24.994ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19515f01-35b6-4e5d-a927-78f72986e949", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54432025-04-30T06:38:24.995ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
54442025-04-30T06:38:24.996ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2039123-5e27-4eb6-bac4-43e80d4e6c1f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54452025-04-30T06:38:24.997ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
54462025-04-30T06:38:24.998ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3661add0-3413-45c3-a6a7-b766be24a37a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54472025-04-30T06:38:24.999ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
54482025-04-30T06:38:25.001ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "617c95e4-4376-4241-9741-71641ef69f4d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54492025-04-30T06:38:25.001ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
54502025-04-30T06:38:25.002ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c8087af-cf0e-421f-bdca-1d513c93c915", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54512025-04-30T06:38:25.002ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
54522025-04-30T06:38:25.003ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8355ca3-3960-4faa-9e6b-0d886ecfd747", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54532025-04-30T06:38:25.004ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
54542025-04-30T06:38:25.005ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a7c8161-0bbf-4c91-b45b-a10ccf3db2df", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54552025-04-30T06:38:25.005ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
54562025-04-30T06:38:25.007ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62a1d2d0-7d83-4de5-b491-11788b2b2f04", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54572025-04-30T06:38:25.007ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
54582025-04-30T06:38:25.008ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce7f1e2e-cee4-41b4-9fd7-c5509d612102", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54592025-04-30T06:38:25.008ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
54602025-04-30T06:38:25.043ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54612025-04-30T06:38:25.043ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54622025-04-30T06:38:25.043ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54632025-04-30T06:38:25.043ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
54642025-04-30T06:38:25.044ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54652025-04-30T06:38:25.046ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43ab31f5-4b58-496f-8376-2bf347930aa0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54662025-04-30T06:38:25.046ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
54672025-04-30T06:38:25.047ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54682025-04-30T06:38:25.054ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54692025-04-30T06:38:25.054ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54702025-04-30T06:38:25.055ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
54712025-04-30T06:38:25.055ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54722025-04-30T06:38:25.057ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b587c441-e043-46c7-a0bb-63ef4b2ff155", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54732025-04-30T06:38:25.057ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
54742025-04-30T06:38:25.057ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54752025-04-30T06:38:25.058ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
54762025-04-30T06:38:25.058ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54772025-04-30T06:38:25.060ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f3a1643-7acd-4141-86f2-bb8cfea3e8f8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54782025-04-30T06:38:25.060ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
54792025-04-30T06:38:25.062ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f513cb88-a402-4ed8-bc49-68acca41416f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54802025-04-30T06:38:25.062ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
54812025-04-30T06:38:25.065ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be3e1ab0-f2c6-4660-9034-325770afb3dd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54822025-04-30T06:38:25.065ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
54832025-04-30T06:38:25.069ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6ade374-e6d7-4c3e-9993-bf6bd8cdcde6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54842025-04-30T06:38:25.069ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
54852025-04-30T06:38:25.071ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7b062ff-c0bc-4ccd-b814-6106b9e7a493", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54862025-04-30T06:38:25.071ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
54872025-04-30T06:38:25.073ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2f6f509-e637-4f04-9886-903ee8b0b0fa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54882025-04-30T06:38:25.073ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
54892025-04-30T06:38:25.074ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad8afe3a-6ded-4f49-a8cc-5f5c154d6dc6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54902025-04-30T06:38:25.074ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
54912025-04-30T06:38:25.076ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72a5b4cc-ba43-4725-bafb-76d5596d5331", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54922025-04-30T06:38:25.076ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
54932025-04-30T06:38:25.077ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df6aae13-75c7-436d-982b-2ba94c242b7d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54942025-04-30T06:38:25.077ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
54952025-04-30T06:38:25.080ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b99817d-8a76-4158-896e-78b40b8a5f0e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54962025-04-30T06:38:25.081ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
54972025-04-30T06:38:25.090ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
54982025-04-30T06:38:25.093ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74a837d8-04a8-417b-9fdf-bf1dd3926585", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
54992025-04-30T06:38:25.093ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
55002025-04-30T06:38:25.095ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82e00dc2-681c-417d-8674-8cd3db689811", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
55012025-04-30T06:38:25.095ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
55022025-04-30T06:38:25.098ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "624780c8-ac5e-4186-ab1a-a4af0ef2a985", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
55032025-04-30T06:38:25.099ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
55042025-04-30T06:38:25.100ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94550fca-a78c-4471-8be6-9c0f42a4ae00", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
55052025-04-30T06:38:25.100ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
55062025-04-30T06:38:25.102ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff41396d-8887-41e9-86f9-213a295298b3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
55072025-04-30T06:38:25.102ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
55082025-04-30T06:38:25.102ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc646a5b-20da-4ac3-bd14-1f7c67b07219", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
55092025-04-30T06:38:25.103ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
55102025-04-30T06:38:25.103ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
55112025-04-30T06:38:25.103ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
55122025-04-30T06:38:25.103ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
55132025-04-30T06:38:25.104ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
55142025-04-30T06:38:25.105ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea12f950-92c8-4de6-ae12-6fc3657a22db", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
55152025-04-30T06:38:25.105ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
55162025-04-30T06:38:25.106ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad4f4093-be1f-4e06-b39c-6093c756d10d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
55172025-04-30T06:38:25.106ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
55182025-04-30T06:38:25.107ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13fffe77-9e72-4e52-bbb9-e3c1f11fa7da", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:24 GMT"} })
55192025-04-30T06:38:25.108ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55202025-04-30T06:38:25.108ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55212025-04-30T06:38:25.108ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55222025-04-30T06:38:25.108ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55232025-04-30T06:38:25.110ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55242025-04-30T06:38:25.110ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
55252025-04-30T06:38:25.110ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
55262025-04-30T06:38:25.117ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55272025-04-30T06:38:25.126ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55282025-04-30T06:38:25.126ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55292025-04-30T06:38:25.126ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55302025-04-30T06:38:25.126ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55312025-04-30T06:38:25.126ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55322025-04-30T06:38:25.126ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55332025-04-30T06:38:25.127ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
55342025-04-30T06:38:25.127ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
55352025-04-30T06:38:25.146ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55362025-04-30T06:38:25.146ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55372025-04-30T06:38:25.146ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55382025-04-30T06:38:25.146ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
55392025-04-30T06:38:25.146ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55402025-04-30T06:38:25.164ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55412025-04-30T06:38:25.169ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55422025-04-30T06:38:25.169ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55432025-04-30T06:38:25.169ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55442025-04-30T06:38:25.170ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55452025-04-30T06:38:25.186ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55462025-04-30T06:38:25.200ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55472025-04-30T06:38:25.201ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55482025-04-30T06:38:25.201ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
55492025-04-30T06:38:25.201ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55502025-04-30T06:38:25.204ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
55512025-04-30T06:38:25.280ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
55522025-04-30T06:38:25.280ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
55532025-04-30T06:38:25.281ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55542025-04-30T06:38:25.281ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
55552025-04-30T06:38:25.282ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55562025-04-30T06:38:25.282ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:51295 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
55572025-04-30T06:38:25.285ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:51295 req_id = 4ef790ee-c93c-4abb-89e1-c6ec7ffa1a63 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55582025-04-30T06:38:25.285ZINFOtest_quotas (dropshot): request completed latency_us = 514 local_addr = [::1]:55655 method = GET remote_addr = [::1]:51295 req_id = 4ef790ee-c93c-4abb-89e1-c6ec7ffa1a63 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55592025-04-30T06:38:25.286ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ef790ee-c93c-4abb-89e1-c6ec7ffa1a63", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
55602025-04-30T06:38:25.289ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55612025-04-30T06:38:25.289ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55622025-04-30T06:38:25.289ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
55632025-04-30T06:38:25.290ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55642025-04-30T06:38:25.290ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55652025-04-30T06:38:25.290ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55662025-04-30T06:38:25.290ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55672025-04-30T06:38:25.291ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55682025-04-30T06:38:25.324ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55692025-04-30T06:38:25.328ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55702025-04-30T06:38:25.328ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55712025-04-30T06:38:25.328ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55722025-04-30T06:38:25.329ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55732025-04-30T06:38:25.384ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
55742025-04-30T06:38:25.388ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
55752025-04-30T06:38:25.388ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
55762025-04-30T06:38:25.389ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 623.997815ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
55772025-04-30T06:38:25.389ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
55782025-04-30T06:38:25.389ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55792025-04-30T06:38:25.389ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55802025-04-30T06:38:25.389ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55812025-04-30T06:38:25.389ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55822025-04-30T06:38:25.390ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55832025-04-30T06:38:25.397ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55842025-04-30T06:38:25.406ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55852025-04-30T06:38:25.406ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55862025-04-30T06:38:25.406ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55872025-04-30T06:38:25.406ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55882025-04-30T06:38:25.415ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
55892025-04-30T06:38:25.416ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
55902025-04-30T06:38:25.416ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 87 89 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50735 req_id = 96fb4668-db1d-4fba-9638-19411a5ec157
55912025-04-30T06:38:25.416ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34697,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50735 req_id = 96fb4668-db1d-4fba-9638-19411a5ec157
55922025-04-30T06:38:25.417ZDEBGtest_quotas (store): query key key = _mgs._tcp
55932025-04-30T06:38:25.417ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55942025-04-30T06:38:25.417ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50735 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 96fb4668-db1d-4fba-9638-19411a5ec157
55952025-04-30T06:38:25.418ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3970, tv_nsec: 216985475 } })
55962025-04-30T06:38:25.418ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
55972025-04-30T06:38:25.418ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 44 86 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e D...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50293 req_id = e6ca7a03-4c58-4cdf-9525-004f0bc07514
55982025-04-30T06:38:25.418ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17542,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50293 req_id = e6ca7a03-4c58-4cdf-9525-004f0bc07514
55992025-04-30T06:38:25.418ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56002025-04-30T06:38:25.418ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50293 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e6ca7a03-4c58-4cdf-9525-004f0bc07514
56012025-04-30T06:38:25.472ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
56022025-04-30T06:38:25.473ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56032025-04-30T06:38:25.473ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56042025-04-30T06:38:25.473ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56052025-04-30T06:38:25.473ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
56062025-04-30T06:38:25.474ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56072025-04-30T06:38:25.474ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5d 3b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ];..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55341 req_id = bd66596e-f12e-4bcc-8384-86cc0225a904
56082025-04-30T06:38:25.474ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23867,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55341 req_id = bd66596e-f12e-4bcc-8384-86cc0225a904
56092025-04-30T06:38:25.474ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
56102025-04-30T06:38:25.475ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23867, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55341 req_id = bd66596e-f12e-4bcc-8384-86cc0225a904
56112025-04-30T06:38:25.475ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
56122025-04-30T06:38:25.475ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
56132025-04-30T06:38:25.475ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
56142025-04-30T06:38:25.477ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "83e658f8-18fa-4dab-9646-43052f2e3bce", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56152025-04-30T06:38:25.477ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
56162025-04-30T06:38:25.478ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56172025-04-30T06:38:25.484ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56182025-04-30T06:38:25.484ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56192025-04-30T06:38:25.484ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
56202025-04-30T06:38:25.484ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56212025-04-30T06:38:25.486ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e8fc81d-8554-4e5b-85cf-8f1bfc49eb79", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56222025-04-30T06:38:25.487ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
56232025-04-30T06:38:25.487ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56242025-04-30T06:38:25.487ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
56252025-04-30T06:38:25.488ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56262025-04-30T06:38:25.496ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44e78c2d-1d47-4ee5-b3a0-9a1c3d2fb8c4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56272025-04-30T06:38:25.496ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
56282025-04-30T06:38:25.513ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dc9482d-a8ba-4c47-8f72-03b0e4c6bf74", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56292025-04-30T06:38:25.513ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
56302025-04-30T06:38:25.515ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aece23c9-147c-449c-825d-0165e1a60ec7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56312025-04-30T06:38:25.515ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
56322025-04-30T06:38:25.518ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1642c725-089b-4f38-a9da-5f7dcc8b7504", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56332025-04-30T06:38:25.518ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
56342025-04-30T06:38:25.521ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63833536-d902-429a-81c1-b927c4b66e41", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56352025-04-30T06:38:25.521ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
56362025-04-30T06:38:25.523ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4f84758-e980-4c76-8396-a0a4350a6fa6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56372025-04-30T06:38:25.523ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
56382025-04-30T06:38:25.524ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fd119d6-501e-4856-9468-d88fba35884f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56392025-04-30T06:38:25.524ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
56402025-04-30T06:38:25.532ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ecb2318-6a1b-4f4f-b5fe-ae863bb3b208", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56412025-04-30T06:38:25.532ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
56422025-04-30T06:38:25.535ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9113bfd-da5a-4178-b8a2-69c97e49e922", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56432025-04-30T06:38:25.535ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
56442025-04-30T06:38:25.535ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d6d979d-f9a4-4f7b-8f40-d99dc7855993", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56452025-04-30T06:38:25.535ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
56462025-04-30T06:38:25.537ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4c843b7-56a6-42ce-8b24-009919d4bdad", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56472025-04-30T06:38:25.538ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
56482025-04-30T06:38:25.541ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
56492025-04-30T06:38:25.543ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b622751-33c1-46b0-937d-26c1327bdfbf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56502025-04-30T06:38:25.543ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
56512025-04-30T06:38:25.546ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "455161d0-0efb-4338-9c5a-933a536f1d12", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56522025-04-30T06:38:25.546ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
56532025-04-30T06:38:25.549ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c596fb08-5007-44c2-80f8-cc86a47e8686", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56542025-04-30T06:38:25.549ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
56552025-04-30T06:38:25.553ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c08e826-f7e5-425a-b1f8-15ea7223a2c8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56562025-04-30T06:38:25.553ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
56572025-04-30T06:38:25.556ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81352789-cfe2-4607-bfa4-b599c6229770", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56582025-04-30T06:38:25.572ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
56592025-04-30T06:38:25.574ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66dc20cf-adb8-441b-ba03-06ed118457a9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56602025-04-30T06:38:25.575ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
56612025-04-30T06:38:25.576ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b2b64e6-a97b-4a64-906b-e60609c4ad00", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56622025-04-30T06:38:25.576ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
56632025-04-30T06:38:25.577ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9073d754-9337-48b9-89d0-b4a7629bca16", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56642025-04-30T06:38:25.577ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
56652025-04-30T06:38:25.578ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2861af82-3aeb-48d1-a06e-c4317ffa1ff0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56662025-04-30T06:38:25.578ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
56672025-04-30T06:38:25.579ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d018531d-002c-4ac8-95b7-4907444d3133", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56682025-04-30T06:38:25.579ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
56692025-04-30T06:38:25.581ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "310e5a66-17e4-45a0-997b-4386f2645e37", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56702025-04-30T06:38:25.581ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
56712025-04-30T06:38:25.583ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09e43d54-d831-4fd9-a7e8-17a75430e84b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56722025-04-30T06:38:25.583ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
56732025-04-30T06:38:25.585ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b1d477f-0207-4370-ab98-407189fac355", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56742025-04-30T06:38:25.585ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
56752025-04-30T06:38:25.587ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed117474-1dc7-4cbc-8c6d-c0d6907adc41", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56762025-04-30T06:38:25.587ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
56772025-04-30T06:38:25.589ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed772a1c-c3a4-4b77-bc7d-eb11cc414a39", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56782025-04-30T06:38:25.589ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
56792025-04-30T06:38:25.592ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "104b9dd7-bb89-4c98-a483-ca5601a2c816", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56802025-04-30T06:38:25.593ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
56812025-04-30T06:38:25.593ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56822025-04-30T06:38:25.593ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56832025-04-30T06:38:25.593ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56842025-04-30T06:38:25.593ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
56852025-04-30T06:38:25.594ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56862025-04-30T06:38:25.602ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56872025-04-30T06:38:25.611ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56882025-04-30T06:38:25.611ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56892025-04-30T06:38:25.611ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
56902025-04-30T06:38:25.611ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56912025-04-30T06:38:25.617ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87551d0f-d98c-4e56-994f-f21fc5bf6033", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56922025-04-30T06:38:25.618ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
56932025-04-30T06:38:25.618ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56942025-04-30T06:38:25.618ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
56952025-04-30T06:38:25.619ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56962025-04-30T06:38:25.622ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcb81222-a09a-4fa3-82be-a0f59f10216b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56972025-04-30T06:38:25.622ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
56982025-04-30T06:38:25.625ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b565f5f9-7874-4733-89ea-7fc4c4038ca4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
56992025-04-30T06:38:25.626ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
57002025-04-30T06:38:25.627ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10619fdc-652e-473a-b76b-d785f9169b59", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57012025-04-30T06:38:25.627ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
57022025-04-30T06:38:25.629ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16c63c27-72d7-4602-9b8a-322354b412c9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57032025-04-30T06:38:25.629ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
57042025-04-30T06:38:25.630ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe76b571-755a-4da0-aa4b-61c30d1ab33f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57052025-04-30T06:38:25.630ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
57062025-04-30T06:38:25.634ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ced273ed-94ae-4a67-8616-da75f91a20e9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57072025-04-30T06:38:25.634ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
57082025-04-30T06:38:25.636ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01af16b8-6f4a-46a8-a171-109930ffba9c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57092025-04-30T06:38:25.636ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
57102025-04-30T06:38:25.639ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b17d887-a181-4ddd-a6f4-2090f82bfbc9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57112025-04-30T06:38:25.639ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
57122025-04-30T06:38:25.641ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fbb581f-d0d3-4776-a3fd-0c61fc198081", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57132025-04-30T06:38:25.641ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
57142025-04-30T06:38:25.643ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cb408bb-9d5a-4907-b975-52c8a0f5e2d0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57152025-04-30T06:38:25.643ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
57162025-04-30T06:38:25.645ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e5380c5-798f-40a0-98d4-f7bc139e8547", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57172025-04-30T06:38:25.646ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
57182025-04-30T06:38:25.649ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4833d75c-f641-4d63-8344-5bcba940cbde", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57192025-04-30T06:38:25.649ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
57202025-04-30T06:38:25.650ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
57212025-04-30T06:38:25.650ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09794275-5cca-45fe-8b95-8f5165bf072e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57222025-04-30T06:38:25.650ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
57232025-04-30T06:38:25.651ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "627f57d6-9da8-49cd-ade0-a4256f1809f9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57242025-04-30T06:38:25.651ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
57252025-04-30T06:38:25.653ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24805b5f-887b-48d7-9002-e3fbea6dacae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57262025-04-30T06:38:25.653ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57272025-04-30T06:38:25.653ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57282025-04-30T06:38:25.653ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57292025-04-30T06:38:25.653ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57302025-04-30T06:38:25.654ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57312025-04-30T06:38:25.663ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57322025-04-30T06:38:25.663ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57332025-04-30T06:38:25.663ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57342025-04-30T06:38:25.663ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57352025-04-30T06:38:25.663ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57362025-04-30T06:38:25.663ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57372025-04-30T06:38:25.663ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57382025-04-30T06:38:25.666ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57392025-04-30T06:38:25.670ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57402025-04-30T06:38:25.670ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57412025-04-30T06:38:25.670ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57422025-04-30T06:38:25.671ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57432025-04-30T06:38:25.721ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
57442025-04-30T06:38:25.721ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
57452025-04-30T06:38:25.721ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57462025-04-30T06:38:25.721ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57472025-04-30T06:38:25.721ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57482025-04-30T06:38:25.721ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
57492025-04-30T06:38:25.722ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57502025-04-30T06:38:25.722ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:49670 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
57512025-04-30T06:38:25.723ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:49670 req_id = fe82c7ac-3313-4699-bbe3-5607e130b46b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57522025-04-30T06:38:25.723ZINFOtest_quotas (dropshot): request completed latency_us = 441 local_addr = [::1]:55655 method = GET remote_addr = [::1]:49670 req_id = fe82c7ac-3313-4699-bbe3-5607e130b46b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57532025-04-30T06:38:25.723ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe82c7ac-3313-4699-bbe3-5607e130b46b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57542025-04-30T06:38:25.725ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57552025-04-30T06:38:25.725ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57562025-04-30T06:38:25.725ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
57572025-04-30T06:38:25.726ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57582025-04-30T06:38:25.726ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57592025-04-30T06:38:25.726ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57602025-04-30T06:38:25.726ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57612025-04-30T06:38:25.727ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57622025-04-30T06:38:25.727ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57632025-04-30T06:38:25.749ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57642025-04-30T06:38:25.749ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57652025-04-30T06:38:25.749ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
57662025-04-30T06:38:25.750ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57672025-04-30T06:38:25.751ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
57682025-04-30T06:38:25.753ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
57692025-04-30T06:38:25.753ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57702025-04-30T06:38:25.757ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57712025-04-30T06:38:25.757ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57722025-04-30T06:38:25.757ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57732025-04-30T06:38:25.758ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57742025-04-30T06:38:25.759ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = ac3eb104-7ebd-47d7-ab4d-1a954dc1379d uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
57752025-04-30T06:38:25.759ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = ac3eb104-7ebd-47d7-ab4d-1a954dc1379d resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
57762025-04-30T06:38:25.759ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = ac3eb104-7ebd-47d7-ab4d-1a954dc1379d roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
57772025-04-30T06:38:25.760ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = ac3eb104-7ebd-47d7-ab4d-1a954dc1379d resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
57782025-04-30T06:38:25.760ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57792025-04-30T06:38:25.760ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
57802025-04-30T06:38:25.761ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57812025-04-30T06:38:25.764ZINFOtest_quotas (dropshot_internal): request completed latency_us = 5496 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = ac3eb104-7ebd-47d7-ab4d-1a954dc1379d response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
57822025-04-30T06:38:25.765ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac3eb104-7ebd-47d7-ab4d-1a954dc1379d", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57832025-04-30T06:38:25.765ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
57842025-04-30T06:38:25.765ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = b9774506-307a-4d35-8f0a-8c5ffb08a1c1 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
57852025-04-30T06:38:25.766ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = b9774506-307a-4d35-8f0a-8c5ffb08a1c1 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
57862025-04-30T06:38:25.766ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = b9774506-307a-4d35-8f0a-8c5ffb08a1c1 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
57872025-04-30T06:38:25.766ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = b9774506-307a-4d35-8f0a-8c5ffb08a1c1 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
57882025-04-30T06:38:25.770ZINFOtest_quotas (dropshot_internal): request completed latency_us = 4352 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = b9774506-307a-4d35-8f0a-8c5ffb08a1c1 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
57892025-04-30T06:38:25.771ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9774506-307a-4d35-8f0a-8c5ffb08a1c1", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
57902025-04-30T06:38:25.772ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
57912025-04-30T06:38:25.772ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
57922025-04-30T06:38:25.772ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
57932025-04-30T06:38:25.826ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
57942025-04-30T06:38:25.827ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
57952025-04-30T06:38:25.827ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 438.86528ms iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
57962025-04-30T06:38:25.828ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
57972025-04-30T06:38:25.828ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57982025-04-30T06:38:25.828ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57992025-04-30T06:38:25.828ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58002025-04-30T06:38:25.828ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58012025-04-30T06:38:25.829ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58022025-04-30T06:38:25.832ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58032025-04-30T06:38:25.836ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
58042025-04-30T06:38:25.836ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58052025-04-30T06:38:25.837ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58062025-04-30T06:38:25.837ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58072025-04-30T06:38:25.840ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
58082025-04-30T06:38:25.840ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
58092025-04-30T06:38:25.842ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5e 0e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ^............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60249 req_id = 99d2b725-bce1-4263-84a6-ede4ae1de49b
58102025-04-30T06:38:25.842ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24078,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60249 req_id = 99d2b725-bce1-4263-84a6-ede4ae1de49b
58112025-04-30T06:38:25.843ZDEBGtest_quotas (store): query key key = _mgs._tcp
58122025-04-30T06:38:25.843ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58132025-04-30T06:38:25.843ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60249 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 99d2b725-bce1-4263-84a6-ede4ae1de49b
58142025-04-30T06:38:25.844ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3970, tv_nsec: 643300699 } })
58152025-04-30T06:38:25.844ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
58162025-04-30T06:38:25.844ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0e c3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51907 req_id = b1873e4a-7371-4282-97f2-a8cf04acec34
58172025-04-30T06:38:25.845ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3779,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51907 req_id = b1873e4a-7371-4282-97f2-a8cf04acec34
58182025-04-30T06:38:25.845ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58192025-04-30T06:38:25.845ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51907 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b1873e4a-7371-4282-97f2-a8cf04acec34
58202025-04-30T06:38:25.965ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
58212025-04-30T06:38:25.966ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58222025-04-30T06:38:25.967ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58232025-04-30T06:38:25.969ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b7 fe 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52195 req_id = 2f3d988c-45f9-4ff0-83ef-8219fedaa611
58242025-04-30T06:38:25.970ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47102,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52195 req_id = 2f3d988c-45f9-4ff0-83ef-8219fedaa611
58252025-04-30T06:38:25.970ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
58262025-04-30T06:38:25.970ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47102, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52195 req_id = 2f3d988c-45f9-4ff0-83ef-8219fedaa611
58272025-04-30T06:38:25.974ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
58282025-04-30T06:38:25.974ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
58292025-04-30T06:38:25.974ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
58302025-04-30T06:38:25.975ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58312025-04-30T06:38:25.975ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58322025-04-30T06:38:25.975ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58332025-04-30T06:38:25.975ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58342025-04-30T06:38:25.976ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
58352025-04-30T06:38:25.993ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fbc1551-16ca-4a83-a87f-da788c105b9e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58362025-04-30T06:38:25.994ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
58372025-04-30T06:38:26.016ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "203f0579-5959-4cdd-a649-7cd6c6554965", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58382025-04-30T06:38:26.016ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
58392025-04-30T06:38:26.024ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81565715-26cd-4b4f-a18e-7d8ae19f137d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58402025-04-30T06:38:26.024ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
58412025-04-30T06:38:26.026ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b300e83-f9a4-41e5-9ccb-92aff15d4c57", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58422025-04-30T06:38:26.026ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
58432025-04-30T06:38:26.027ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58442025-04-30T06:38:26.027ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58452025-04-30T06:38:26.027ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58462025-04-30T06:38:26.027ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
58472025-04-30T06:38:26.029ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58482025-04-30T06:38:26.033ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58492025-04-30T06:38:26.045ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58502025-04-30T06:38:26.045ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58512025-04-30T06:38:26.045ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
58522025-04-30T06:38:26.046ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58532025-04-30T06:38:26.048ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10d481c1-ea2d-47e3-8089-860d0c5d3849", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58542025-04-30T06:38:26.049ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
58552025-04-30T06:38:26.049ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58562025-04-30T06:38:26.049ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
58572025-04-30T06:38:26.050ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58582025-04-30T06:38:26.056ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7463f323-5069-4ce4-981d-d88801c22455", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58592025-04-30T06:38:26.057ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
58602025-04-30T06:38:26.060ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19ce2e31-f060-4931-9ca0-91b19af71847", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58612025-04-30T06:38:26.060ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
58622025-04-30T06:38:26.064ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ead8a6ea-9675-4be4-8c9f-f6bc78b8caba", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58632025-04-30T06:38:26.064ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
58642025-04-30T06:38:26.066ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e203799f-0bb6-426b-a5a5-9313bbc46292", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58652025-04-30T06:38:26.068ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
58662025-04-30T06:38:26.073ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e64559d-e6b8-4e52-8bf4-2f6f24d78db9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58672025-04-30T06:38:26.073ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
58682025-04-30T06:38:26.096ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d068374-e63d-423f-aa67-0a94db6df46b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58692025-04-30T06:38:26.097ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
58702025-04-30T06:38:26.098ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "571a3a09-c99d-4315-a7b4-24b20be123cb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58712025-04-30T06:38:26.098ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
58722025-04-30T06:38:26.100ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5006814-a9f9-45a0-bd35-55f37a64b6f8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58732025-04-30T06:38:26.101ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
58742025-04-30T06:38:26.106ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58752025-04-30T06:38:26.106ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58762025-04-30T06:38:26.106ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
58772025-04-30T06:38:26.108ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
58782025-04-30T06:38:26.110ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
58792025-04-30T06:38:26.115ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
58802025-04-30T06:38:26.115ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
58812025-04-30T06:38:26.119ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
58822025-04-30T06:38:26.119ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58832025-04-30T06:38:26.121ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d54b2cc-1f81-4733-9e27-6f04571f500f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58842025-04-30T06:38:26.121ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
58852025-04-30T06:38:26.124ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "217d3872-40ba-4c99-b48f-ad78b671b7e1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58862025-04-30T06:38:26.125ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
58872025-04-30T06:38:26.128ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6773abc8-d267-4f07-8072-d719212f8d8c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58882025-04-30T06:38:26.128ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
58892025-04-30T06:38:26.131ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a00448e3-a36a-42ea-84c4-475f718a78a9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58902025-04-30T06:38:26.131ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
58912025-04-30T06:38:26.133ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ce5574c-b3b9-4ac0-9df4-519d627f4128", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58922025-04-30T06:38:26.133ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
58932025-04-30T06:38:26.136ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ffff23e-4845-4d93-ba43-b734b38fc596", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58942025-04-30T06:38:26.136ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
58952025-04-30T06:38:26.150ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfb3c6da-b4fa-48f1-9a7d-1f10a0a12530", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58962025-04-30T06:38:26.150ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
58972025-04-30T06:38:26.152ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c090299-425c-48fb-8c86-dc068243672b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
58982025-04-30T06:38:26.152ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
58992025-04-30T06:38:26.153ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c48ead6-9f0f-4d66-a880-60a70014e7c5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59002025-04-30T06:38:26.153ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
59012025-04-30T06:38:26.154ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8253a7b3-01d1-4cc0-94a5-853724221850", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59022025-04-30T06:38:26.154ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
59032025-04-30T06:38:26.155ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5acca0f6-13c9-4b01-b020-32346ccec112", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59042025-04-30T06:38:26.156ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
59052025-04-30T06:38:26.158ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25122a3e-587a-4258-89b7-f5b757d6321f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59062025-04-30T06:38:26.158ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
59072025-04-30T06:38:26.160ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "786fb4ae-e0aa-49f4-bf56-43b2d9e18a9e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59082025-04-30T06:38:26.160ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
59092025-04-30T06:38:26.161ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59102025-04-30T06:38:26.162ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59112025-04-30T06:38:26.162ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59122025-04-30T06:38:26.162ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
59132025-04-30T06:38:26.163ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59142025-04-30T06:38:26.167ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59152025-04-30T06:38:26.173ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59162025-04-30T06:38:26.173ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59172025-04-30T06:38:26.173ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
59182025-04-30T06:38:26.173ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59192025-04-30T06:38:26.174ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f382fbe1-143e-4294-9b9d-cdd7772a02b7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59202025-04-30T06:38:26.174ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
59212025-04-30T06:38:26.177ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59222025-04-30T06:38:26.177ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
59232025-04-30T06:38:26.177ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59242025-04-30T06:38:26.178ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35ab5d64-2b1c-446f-a405-5f8e9db1fd2c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59252025-04-30T06:38:26.179ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
59262025-04-30T06:38:26.190ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d103b27d-c8ad-4878-9fbc-6d229a348849", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59272025-04-30T06:38:26.190ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
59282025-04-30T06:38:26.192ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5883b97-f9dc-4743-933d-6ffefd51a69d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59292025-04-30T06:38:26.193ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
59302025-04-30T06:38:26.193ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9acc34cc-4747-4b4b-bce9-0499749a8ff0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59312025-04-30T06:38:26.194ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
59322025-04-30T06:38:26.197ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55de6311-9062-46d9-ad57-cda32964759b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59332025-04-30T06:38:26.198ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
59342025-04-30T06:38:26.199ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ac1e5ea-c724-4380-bf26-5e6440691007", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59352025-04-30T06:38:26.199ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
59362025-04-30T06:38:26.202ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eab52b5-f1b2-46c8-bbed-56fc10ce6660", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59372025-04-30T06:38:26.202ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
59382025-04-30T06:38:26.204ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90e861aa-6ffb-42fe-8f78-13dcb686f128", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59392025-04-30T06:38:26.204ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
59402025-04-30T06:38:26.207ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
59412025-04-30T06:38:26.210ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8785d96-d781-43f2-8a5b-93c6c5abf103", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59422025-04-30T06:38:26.211ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
59432025-04-30T06:38:26.215ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b77cee1-2adb-4239-8dd8-b0f0feba9104", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59442025-04-30T06:38:26.215ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
59452025-04-30T06:38:26.218ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "991ecc03-ca6b-4a3d-8ba3-210986e36f13", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59462025-04-30T06:38:26.218ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
59472025-04-30T06:38:26.220ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c63b9fbd-544b-4544-8f19-93d3dc387549", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59482025-04-30T06:38:26.220ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
59492025-04-30T06:38:26.224ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
59502025-04-30T06:38:26.224ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab4d258f-0d20-40bd-8cb2-bfb96e2efb21", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59512025-04-30T06:38:26.224ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
59522025-04-30T06:38:26.227ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1270723-1961-4fbb-86a5-5098ddf36f6a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59532025-04-30T06:38:26.227ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
59542025-04-30T06:38:26.228ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe846a14-55bb-4be0-8e85-319753044d18", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59552025-04-30T06:38:26.228ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
59562025-04-30T06:38:26.229ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12654e0f-230b-49c3-8fd1-9cdd270450bc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59572025-04-30T06:38:26.229ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
59582025-04-30T06:38:26.230ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35bb2910-2938-42c7-adc1-747b6763ba04", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59592025-04-30T06:38:26.230ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
59602025-04-30T06:38:26.231ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48097af2-25be-4b47-a122-42429c0770c5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:25 GMT"} })
59612025-04-30T06:38:26.231ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59622025-04-30T06:38:26.231ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59632025-04-30T06:38:26.231ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59642025-04-30T06:38:26.231ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59652025-04-30T06:38:26.232ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59662025-04-30T06:38:26.234ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59672025-04-30T06:38:26.241ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59682025-04-30T06:38:26.241ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59692025-04-30T06:38:26.241ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59702025-04-30T06:38:26.241ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59712025-04-30T06:38:26.241ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59722025-04-30T06:38:26.242ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59732025-04-30T06:38:26.244ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59742025-04-30T06:38:26.249ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59752025-04-30T06:38:26.249ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59762025-04-30T06:38:26.249ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59772025-04-30T06:38:26.249ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59782025-04-30T06:38:26.340ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
59792025-04-30T06:38:26.341ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
59802025-04-30T06:38:26.342ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59812025-04-30T06:38:26.342ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59822025-04-30T06:38:26.342ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59832025-04-30T06:38:26.342ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
59842025-04-30T06:38:26.343ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59852025-04-30T06:38:26.343ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:63342 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
59862025-04-30T06:38:26.345ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:63342 req_id = d8d9f062-5dae-48eb-84ad-c6579abc7279 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59872025-04-30T06:38:26.346ZINFOtest_quotas (dropshot): request completed latency_us = 1254 local_addr = [::1]:55655 method = GET remote_addr = [::1]:63342 req_id = d8d9f062-5dae-48eb-84ad-c6579abc7279 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59882025-04-30T06:38:26.347ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59892025-04-30T06:38:26.413ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59902025-04-30T06:38:26.413ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59912025-04-30T06:38:26.413ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
59922025-04-30T06:38:26.413ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59932025-04-30T06:38:26.414ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8d9f062-5dae-48eb-84ad-c6579abc7279", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
59942025-04-30T06:38:26.419ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59952025-04-30T06:38:26.419ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59962025-04-30T06:38:26.419ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
59972025-04-30T06:38:26.421ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59982025-04-30T06:38:26.421ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59992025-04-30T06:38:26.421ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60002025-04-30T06:38:26.421ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60012025-04-30T06:38:26.423ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60022025-04-30T06:38:26.428ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60032025-04-30T06:38:26.428ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
60042025-04-30T06:38:26.428ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60052025-04-30T06:38:26.429ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60062025-04-30T06:38:26.442ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60072025-04-30T06:38:26.443ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60082025-04-30T06:38:26.443ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60092025-04-30T06:38:26.453ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60102025-04-30T06:38:26.511ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
60112025-04-30T06:38:26.511ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
60122025-04-30T06:38:26.511ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
60132025-04-30T06:38:26.511ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 683.870808ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
60142025-04-30T06:38:26.512ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
60152025-04-30T06:38:26.513ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
60162025-04-30T06:38:26.513ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60172025-04-30T06:38:26.513ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60182025-04-30T06:38:26.513ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60192025-04-30T06:38:26.514ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60202025-04-30T06:38:26.528ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60212025-04-30T06:38:26.534ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60222025-04-30T06:38:26.535ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60232025-04-30T06:38:26.535ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60242025-04-30T06:38:26.540ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60252025-04-30T06:38:26.563ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60262025-04-30T06:38:26.563ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60272025-04-30T06:38:26.563ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60282025-04-30T06:38:26.563ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
60292025-04-30T06:38:26.564ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60302025-04-30T06:38:26.564ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
60312025-04-30T06:38:26.564ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
60322025-04-30T06:38:26.565ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b2 87 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52656 req_id = 345d81c4-f074-4000-b7da-9e3575667413
60332025-04-30T06:38:26.565ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45703,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52656 req_id = 345d81c4-f074-4000-b7da-9e3575667413
60342025-04-30T06:38:26.565ZDEBGtest_quotas (store): query key key = _mgs._tcp
60352025-04-30T06:38:26.565ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60362025-04-30T06:38:26.565ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52656 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 345d81c4-f074-4000-b7da-9e3575667413
60372025-04-30T06:38:26.566ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3971, tv_nsec: 365357130 } })
60382025-04-30T06:38:26.566ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
60392025-04-30T06:38:26.567ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3d 02 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e =...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64364 req_id = d72cbf13-295d-423c-9078-0b6bacf11def
60402025-04-30T06:38:26.567ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15618,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64364 req_id = d72cbf13-295d-423c-9078-0b6bacf11def
60412025-04-30T06:38:26.567ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60422025-04-30T06:38:26.567ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64364 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d72cbf13-295d-423c-9078-0b6bacf11def
60432025-04-30T06:38:26.567ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60442025-04-30T06:38:26.579ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60452025-04-30T06:38:26.579ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60462025-04-30T06:38:26.579ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
60472025-04-30T06:38:26.579ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60482025-04-30T06:38:26.665ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
60492025-04-30T06:38:26.667ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 78 9b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c x............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59262 req_id = b4ef2866-8a6f-41b3-be96-e3d9f483cb9d
60502025-04-30T06:38:26.667ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30875,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59262 req_id = b4ef2866-8a6f-41b3-be96-e3d9f483cb9d
60512025-04-30T06:38:26.669ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
60522025-04-30T06:38:26.669ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30875, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59262 req_id = b4ef2866-8a6f-41b3-be96-e3d9f483cb9d
60532025-04-30T06:38:26.669ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
60542025-04-30T06:38:26.669ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
60552025-04-30T06:38:26.669ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
60562025-04-30T06:38:26.669ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60572025-04-30T06:38:26.669ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
60582025-04-30T06:38:26.669ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60592025-04-30T06:38:26.672ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4733e853-a909-413d-9de3-aa66db6c13ca", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60602025-04-30T06:38:26.672ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
60612025-04-30T06:38:26.676ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f161810-34a9-4ef1-a750-3f72f6a38466", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60622025-04-30T06:38:26.676ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
60632025-04-30T06:38:26.679ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f8ebc72-8988-4d4c-b8ac-9b3a8b611833", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60642025-04-30T06:38:26.679ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
60652025-04-30T06:38:26.682ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f11848ad-43f0-4978-8e41-2a84c072b154", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60662025-04-30T06:38:26.682ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
60672025-04-30T06:38:26.685ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7dcb8eb-f4fd-47e3-a624-7bcbea6a1e45", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60682025-04-30T06:38:26.685ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
60692025-04-30T06:38:26.688ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f665bb2-8d4a-4876-8511-7695d49708bb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60702025-04-30T06:38:26.688ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
60712025-04-30T06:38:26.690ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce252bde-77d0-4186-b93a-8b59c1d49e6a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60722025-04-30T06:38:26.691ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
60732025-04-30T06:38:26.695ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b479862-e58e-4539-9354-3b8c0bfc9183", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60742025-04-30T06:38:26.695ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
60752025-04-30T06:38:26.709ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
60762025-04-30T06:38:26.710ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e69e672-19aa-4c9c-a4f5-fb6d763ff7f4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60772025-04-30T06:38:26.710ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
60782025-04-30T06:38:26.712ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de089535-4449-455d-bc45-933e3459b01d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60792025-04-30T06:38:26.713ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
60802025-04-30T06:38:26.714ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95a84da9-2d01-4926-a232-72ad0e3fb714", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60812025-04-30T06:38:26.714ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
60822025-04-30T06:38:26.715ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c730d2-387f-4427-8964-c872ce70c232", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60832025-04-30T06:38:26.715ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
60842025-04-30T06:38:26.717ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c322b43-1dee-4673-b307-1ef83f058478", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60852025-04-30T06:38:26.717ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
60862025-04-30T06:38:26.727ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a152ca3b-d6e1-4b49-bed9-b0cdcd29a1c8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60872025-04-30T06:38:26.729ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
60882025-04-30T06:38:26.735ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68a4b1b6-e672-424a-81d2-55a67e63e482", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60892025-04-30T06:38:26.735ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
60902025-04-30T06:38:26.740ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e18c256f-c1d6-4999-8ede-560f3acefe65", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60912025-04-30T06:38:26.742ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
60922025-04-30T06:38:26.746ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "403e798f-a41b-4950-88b0-f6c9ad18a68b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60932025-04-30T06:38:26.746ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
60942025-04-30T06:38:26.749ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1921a85c-70d7-405e-9ed6-05a851822467", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60952025-04-30T06:38:26.749ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
60962025-04-30T06:38:26.752ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "548186af-aa8b-4ee0-873b-d1e8bbb54178", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60972025-04-30T06:38:26.752ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
60982025-04-30T06:38:26.753ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e56af57e-d04e-4643-a7d2-76af4475a98c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
60992025-04-30T06:38:26.753ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
61002025-04-30T06:38:26.754ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd7e769b-7161-4807-85d1-69a90121dfb6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61012025-04-30T06:38:26.754ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
61022025-04-30T06:38:26.755ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f3aed78-38da-4490-99ba-b9b55f9351dc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61032025-04-30T06:38:26.755ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
61042025-04-30T06:38:26.756ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcfbcd5c-2562-4135-a93b-da5d2a794343", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61052025-04-30T06:38:26.757ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
61062025-04-30T06:38:26.758ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea5f74fc-42a0-4945-8e77-10f68e6b5570", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61072025-04-30T06:38:26.758ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
61082025-04-30T06:38:26.760ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8a37010-ac3d-406d-a134-0685f41fcddc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61092025-04-30T06:38:26.760ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
61102025-04-30T06:38:26.761ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61112025-04-30T06:38:26.761ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61122025-04-30T06:38:26.764ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61132025-04-30T06:38:26.764ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
61142025-04-30T06:38:26.765ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61152025-04-30T06:38:26.768ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cae2afe8-2152-4e5e-9cba-aeec807cc7e6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61162025-04-30T06:38:26.768ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
61172025-04-30T06:38:26.769ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61182025-04-30T06:38:26.780ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61192025-04-30T06:38:26.780ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61202025-04-30T06:38:26.780ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
61212025-04-30T06:38:26.781ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61222025-04-30T06:38:26.784ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61232025-04-30T06:38:26.784ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
61242025-04-30T06:38:26.785ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61252025-04-30T06:38:26.786ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91b04fb4-9bc0-4b17-915a-ad69adb0e8a5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61262025-04-30T06:38:26.786ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
61272025-04-30T06:38:26.790ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ba41285-0c95-4d6c-af79-203d07aed5a7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61282025-04-30T06:38:26.791ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
61292025-04-30T06:38:26.794ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "613522b3-0c27-49a7-8963-0b32fecbee31", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61302025-04-30T06:38:26.795ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
61312025-04-30T06:38:26.797ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0961627c-595b-4e57-92cb-fbe650b70eef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61322025-04-30T06:38:26.797ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
61332025-04-30T06:38:26.798ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edca3a6c-6e48-480e-894d-b412b24fdfe5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61342025-04-30T06:38:26.798ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
61352025-04-30T06:38:26.800ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c715898-1648-4b73-8d38-5c9c1802be8d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61362025-04-30T06:38:26.800ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
61372025-04-30T06:38:26.801ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d05e5c7c-6502-4f52-a600-86a4d9cf914f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61382025-04-30T06:38:26.801ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
61392025-04-30T06:38:26.803ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8f5ced4-565d-4bbc-835d-cc8abc898a58", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61402025-04-30T06:38:26.803ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
61412025-04-30T06:38:26.808ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfef1c20-c779-4956-8630-d5d49377f22c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61422025-04-30T06:38:26.809ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
61432025-04-30T06:38:26.822ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeeea8b9-6a70-4ee7-bb38-cb89623bbb9c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61442025-04-30T06:38:26.822ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
61452025-04-30T06:38:26.822ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
61462025-04-30T06:38:26.825ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3410144a-b6ec-4c4a-8729-72c32eb3aef4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61472025-04-30T06:38:26.826ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
61482025-04-30T06:38:26.828ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05a27aa1-cf47-4069-8ada-68731abf0747", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61492025-04-30T06:38:26.828ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
61502025-04-30T06:38:26.830ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e69dc50-b1d9-4ee6-bcc3-a6fee85eeb7b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61512025-04-30T06:38:26.830ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
61522025-04-30T06:38:26.834ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c757ed8b-dcaa-4237-913e-251bfb5bcf54", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61532025-04-30T06:38:26.835ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
61542025-04-30T06:38:26.840ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8015edf3-d202-48ac-9085-032698294b36", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61552025-04-30T06:38:26.840ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
61562025-04-30T06:38:26.847ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8b8c0d2-54c9-4c8e-ba55-b06eb2e530b0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61572025-04-30T06:38:26.848ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
61582025-04-30T06:38:26.849ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f791a52f-ab9c-4f11-9c2b-9ad4bdcee74d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61592025-04-30T06:38:26.850ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
61602025-04-30T06:38:26.850ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61612025-04-30T06:38:26.850ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61622025-04-30T06:38:26.858ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61632025-04-30T06:38:26.859ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61642025-04-30T06:38:26.859ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61652025-04-30T06:38:26.859ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61662025-04-30T06:38:26.859ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95346efb-0a16-4a0f-84c7-c1e34109f13e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61672025-04-30T06:38:26.859ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
61682025-04-30T06:38:26.861ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1709e14b-66d9-40cd-80a4-ccb4f9e69479", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
61692025-04-30T06:38:26.861ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61702025-04-30T06:38:26.861ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61712025-04-30T06:38:26.861ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61722025-04-30T06:38:26.861ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61732025-04-30T06:38:26.864ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61742025-04-30T06:38:26.871ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61752025-04-30T06:38:26.877ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61762025-04-30T06:38:26.877ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61772025-04-30T06:38:26.877ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61782025-04-30T06:38:26.877ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61792025-04-30T06:38:26.877ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61802025-04-30T06:38:26.878ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61812025-04-30T06:38:26.879ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61822025-04-30T06:38:26.879ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61832025-04-30T06:38:26.879ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61842025-04-30T06:38:26.879ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
61852025-04-30T06:38:26.879ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61862025-04-30T06:38:26.881ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61872025-04-30T06:38:26.896ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61882025-04-30T06:38:26.896ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61892025-04-30T06:38:26.897ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61902025-04-30T06:38:26.898ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61912025-04-30T06:38:26.899ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61922025-04-30T06:38:26.909ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61932025-04-30T06:38:26.909ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61942025-04-30T06:38:26.909ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
61952025-04-30T06:38:26.910ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61962025-04-30T06:38:26.991ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
61972025-04-30T06:38:26.991ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
61982025-04-30T06:38:26.992ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:33630 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
61992025-04-30T06:38:26.993ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:33630 req_id = b04f9896-c101-4f5a-a874-23159fb31828 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62002025-04-30T06:38:26.994ZINFOtest_quotas (dropshot): request completed latency_us = 576 local_addr = [::1]:55655 method = GET remote_addr = [::1]:33630 req_id = b04f9896-c101-4f5a-a874-23159fb31828 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62012025-04-30T06:38:26.994ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62022025-04-30T06:38:26.994ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
62032025-04-30T06:38:26.995ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62042025-04-30T06:38:26.995ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b04f9896-c101-4f5a-a874-23159fb31828", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62052025-04-30T06:38:26.997ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62062025-04-30T06:38:26.997ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62072025-04-30T06:38:26.998ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
62082025-04-30T06:38:26.999ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62092025-04-30T06:38:26.999ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62102025-04-30T06:38:26.999ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62112025-04-30T06:38:26.999ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62122025-04-30T06:38:27.000ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62132025-04-30T06:38:27.013ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62142025-04-30T06:38:27.019ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62152025-04-30T06:38:27.019ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62162025-04-30T06:38:27.019ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62172025-04-30T06:38:27.020ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62182025-04-30T06:38:27.061ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
62192025-04-30T06:38:27.062ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
62202025-04-30T06:38:27.062ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 550.192442ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
62212025-04-30T06:38:27.062ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
62222025-04-30T06:38:27.062ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62232025-04-30T06:38:27.062ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62242025-04-30T06:38:27.062ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62252025-04-30T06:38:27.062ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62262025-04-30T06:38:27.063ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62272025-04-30T06:38:27.063ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
62282025-04-30T06:38:27.066ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62292025-04-30T06:38:27.074ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62302025-04-30T06:38:27.074ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62312025-04-30T06:38:27.074ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62322025-04-30T06:38:27.074ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62332025-04-30T06:38:27.078ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
62342025-04-30T06:38:27.078ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
62352025-04-30T06:38:27.079ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 97 16 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52324 req_id = 3e5f1a63-438c-4995-8e09-40569fda8b37
62362025-04-30T06:38:27.079ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38678,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52324 req_id = 3e5f1a63-438c-4995-8e09-40569fda8b37
62372025-04-30T06:38:27.079ZDEBGtest_quotas (store): query key key = _mgs._tcp
62382025-04-30T06:38:27.079ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62392025-04-30T06:38:27.079ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52324 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3e5f1a63-438c-4995-8e09-40569fda8b37
62402025-04-30T06:38:27.080ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3971, tv_nsec: 878950431 } })
62412025-04-30T06:38:27.080ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
62422025-04-30T06:38:27.080ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 93 5f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ._..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56262 req_id = a61fda1b-686c-4aa9-9748-49c07b50ac44
62432025-04-30T06:38:27.080ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37727,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56262 req_id = a61fda1b-686c-4aa9-9748-49c07b50ac44
62442025-04-30T06:38:27.080ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62452025-04-30T06:38:27.080ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56262 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a61fda1b-686c-4aa9-9748-49c07b50ac44
62462025-04-30T06:38:27.134ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
62472025-04-30T06:38:27.135ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62482025-04-30T06:38:27.135ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62492025-04-30T06:38:27.135ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62502025-04-30T06:38:27.135ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
62512025-04-30T06:38:27.135ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62522025-04-30T06:38:27.136ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62532025-04-30T06:38:27.136ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62542025-04-30T06:38:27.136ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
62552025-04-30T06:38:27.137ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
62562025-04-30T06:38:27.137ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5f 35 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c _5..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63018 req_id = f633b1cc-f737-471d-bc34-3fa8ffbabecf
62572025-04-30T06:38:27.137ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24373,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63018 req_id = f633b1cc-f737-471d-bc34-3fa8ffbabecf
62582025-04-30T06:38:27.138ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
62592025-04-30T06:38:27.138ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24373, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63018 req_id = f633b1cc-f737-471d-bc34-3fa8ffbabecf
62602025-04-30T06:38:27.138ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
62612025-04-30T06:38:27.138ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
62622025-04-30T06:38:27.138ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
62632025-04-30T06:38:27.139ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62642025-04-30T06:38:27.144ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62652025-04-30T06:38:27.144ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62662025-04-30T06:38:27.144ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
62672025-04-30T06:38:27.145ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62682025-04-30T06:38:27.147ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f20c3a7-d117-4a7c-b253-081abef25be9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62692025-04-30T06:38:27.148ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
62702025-04-30T06:38:27.149ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62712025-04-30T06:38:27.149ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
62722025-04-30T06:38:27.149ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62732025-04-30T06:38:27.150ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
62742025-04-30T06:38:27.150ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
62752025-04-30T06:38:27.152ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0d331c6-ee82-4372-b39a-c91c628a10ce", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62762025-04-30T06:38:27.153ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
62772025-04-30T06:38:27.153ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
62782025-04-30T06:38:27.154ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62792025-04-30T06:38:27.159ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaabf7aa-d9c0-45a5-99a6-7b7d377ee514", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62802025-04-30T06:38:27.159ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
62812025-04-30T06:38:27.164ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9620a4b4-63f6-4212-828d-1b3f137ec489", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62822025-04-30T06:38:27.164ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
62832025-04-30T06:38:27.168ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8f3cf8e-2fb5-4a4b-b99d-4d5e1e5502d0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62842025-04-30T06:38:27.169ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
62852025-04-30T06:38:27.173ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7930517-d415-4076-8e1b-9e81d3e0890d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62862025-04-30T06:38:27.173ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
62872025-04-30T06:38:27.178ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cd36e1a-cf77-434a-b8ff-21694115c5bf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62882025-04-30T06:38:27.178ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
62892025-04-30T06:38:27.182ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
62902025-04-30T06:38:27.184ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22eff5ee-1f49-4013-b420-cce3b13eb54f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62912025-04-30T06:38:27.184ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
62922025-04-30T06:38:27.186ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "255fe7df-c7db-45a9-af78-22b351432b68", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62932025-04-30T06:38:27.187ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
62942025-04-30T06:38:27.188ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb2d8e15-cd6d-41ae-814b-d7e36e965225", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62952025-04-30T06:38:27.189ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
62962025-04-30T06:38:27.192ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "124edda7-e278-46a1-acb9-fb34bc711934", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62972025-04-30T06:38:27.194ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
62982025-04-30T06:38:27.196ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb9f3813-5a76-4e07-b2c1-2c6c1b9694d0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
62992025-04-30T06:38:27.197ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
63002025-04-30T06:38:27.198ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcae839d-6428-4bf2-b852-d38b2d011763", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63012025-04-30T06:38:27.198ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
63022025-04-30T06:38:27.200ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00b6248c-e822-4b62-b964-19b91cd2864f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63032025-04-30T06:38:27.200ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
63042025-04-30T06:38:27.201ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "114f5dea-daff-470b-bfac-1e4f090265ba", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63052025-04-30T06:38:27.202ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
63062025-04-30T06:38:27.204ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "109be8b0-4a32-4bb0-8668-712d8f306da7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63072025-04-30T06:38:27.204ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
63082025-04-30T06:38:27.213ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5663ce2a-88f1-4753-9fdf-e37c27f5d472", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63092025-04-30T06:38:27.213ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
63102025-04-30T06:38:27.216ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39062e8c-cab6-4313-93eb-6a3b4ced9f34", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63112025-04-30T06:38:27.218ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
63122025-04-30T06:38:27.224ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6639a85b-9ee9-43fd-9e61-f9f234a955b8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63132025-04-30T06:38:27.224ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
63142025-04-30T06:38:27.226ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd784225-2b77-412b-a56e-97028b37aff2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63152025-04-30T06:38:27.226ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
63162025-04-30T06:38:27.228ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93d935d8-1ceb-479b-a8f2-c8a349233221", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63172025-04-30T06:38:27.228ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
63182025-04-30T06:38:27.230ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6a9b1ce-a70a-4381-926b-73d05d0460bc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63192025-04-30T06:38:27.231ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
63202025-04-30T06:38:27.232ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e150c70c-9dc8-48f5-92e3-ef9b69bdabb7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63212025-04-30T06:38:27.233ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
63222025-04-30T06:38:27.233ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63232025-04-30T06:38:27.233ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63242025-04-30T06:38:27.233ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63252025-04-30T06:38:27.233ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
63262025-04-30T06:38:27.235ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63272025-04-30T06:38:27.237ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
63282025-04-30T06:38:27.238ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbf96028-a1d5-4fdd-9880-10c4654addf7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63292025-04-30T06:38:27.246ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
63302025-04-30T06:38:27.247ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63312025-04-30T06:38:27.255ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63322025-04-30T06:38:27.255ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63332025-04-30T06:38:27.255ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
63342025-04-30T06:38:27.255ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63352025-04-30T06:38:27.258ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "445b02a4-2b57-4081-ae53-2c7262dc699c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63362025-04-30T06:38:27.258ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
63372025-04-30T06:38:27.258ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63382025-04-30T06:38:27.258ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
63392025-04-30T06:38:27.258ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63402025-04-30T06:38:27.261ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b4cf77f-8db0-4676-a5c4-0eb7840ba96c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63412025-04-30T06:38:27.261ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
63422025-04-30T06:38:27.263ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73aa8a7a-59b7-41f5-a1d8-d2732e4bb1ce", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63432025-04-30T06:38:27.263ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
63442025-04-30T06:38:27.265ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa595c4b-65ca-4c74-8a41-ca4c27c50e56", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63452025-04-30T06:38:27.265ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
63462025-04-30T06:38:27.269ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "617a3306-a2f1-4857-8774-086fee7b13ad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63472025-04-30T06:38:27.269ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
63482025-04-30T06:38:27.273ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93b07251-fd33-4947-acef-6c9796d7752c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63492025-04-30T06:38:27.273ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
63502025-04-30T06:38:27.275ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "101a1cee-23aa-49ef-939f-2984a4c0e173", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63512025-04-30T06:38:27.275ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
63522025-04-30T06:38:27.277ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1894279d-5651-43dc-a1c4-7e20fd63c78a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63532025-04-30T06:38:27.277ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
63542025-04-30T06:38:27.284ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "389e8260-d70a-49ca-8e5d-a4447a868303", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63552025-04-30T06:38:27.284ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
63562025-04-30T06:38:27.286ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "520243fc-1bd1-4cf3-89fe-5bfa2565c900", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63572025-04-30T06:38:27.286ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
63582025-04-30T06:38:27.290ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "981404ca-2648-43dd-be7f-cd2658f5c24f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63592025-04-30T06:38:27.290ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
63602025-04-30T06:38:27.291ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
63612025-04-30T06:38:27.295ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62225f63-b9ad-4d9b-a744-4532821f7617", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63622025-04-30T06:38:27.296ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
63632025-04-30T06:38:27.299ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1f62536-5f09-49d4-9198-183d5392959e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63642025-04-30T06:38:27.299ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
63652025-04-30T06:38:27.303ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "265550a7-8b40-4f45-a357-bde3d3252a0e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63662025-04-30T06:38:27.303ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
63672025-04-30T06:38:27.307ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7555f6da-bdf6-490b-b4af-b6ccdbf8a2ed", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63682025-04-30T06:38:27.307ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
63692025-04-30T06:38:27.309ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05b9fe3a-f592-4d66-8df6-2225c66fb982", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63702025-04-30T06:38:27.309ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
63712025-04-30T06:38:27.311ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a69f200b-2373-4f25-a4f5-cb452020ff20", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63722025-04-30T06:38:27.311ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
63732025-04-30T06:38:27.312ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "209220ed-5381-4c8f-ad6e-66845d5dea40", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63742025-04-30T06:38:27.312ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
63752025-04-30T06:38:27.313ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fba18a73-b538-403d-b41f-c98d6836ca4d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63762025-04-30T06:38:27.313ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
63772025-04-30T06:38:27.314ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05e2a404-cd30-4351-9d32-eb257a1c5836", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63782025-04-30T06:38:27.314ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
63792025-04-30T06:38:27.315ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9209ede4-8b44-4e76-b1d4-a0698df87528", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:26 GMT"} })
63802025-04-30T06:38:27.315ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63812025-04-30T06:38:27.315ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63822025-04-30T06:38:27.315ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63832025-04-30T06:38:27.315ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63842025-04-30T06:38:27.316ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63852025-04-30T06:38:27.318ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63862025-04-30T06:38:27.324ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63872025-04-30T06:38:27.324ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63882025-04-30T06:38:27.324ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63892025-04-30T06:38:27.324ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63902025-04-30T06:38:27.324ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63912025-04-30T06:38:27.325ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63922025-04-30T06:38:27.327ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63932025-04-30T06:38:27.335ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63942025-04-30T06:38:27.335ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63952025-04-30T06:38:27.335ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63962025-04-30T06:38:27.336ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63972025-04-30T06:38:27.400ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
63982025-04-30T06:38:27.400ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
63992025-04-30T06:38:27.400ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64002025-04-30T06:38:27.400ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64012025-04-30T06:38:27.401ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64022025-04-30T06:38:27.401ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
64032025-04-30T06:38:27.401ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64042025-04-30T06:38:27.401ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:33477 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
64052025-04-30T06:38:27.402ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:33477 req_id = 2785ad0c-7dc5-4c5a-9f5a-af6f186e5993 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64062025-04-30T06:38:27.402ZINFOtest_quotas (dropshot): request completed latency_us = 476 local_addr = [::1]:55655 method = GET remote_addr = [::1]:33477 req_id = 2785ad0c-7dc5-4c5a-9f5a-af6f186e5993 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64072025-04-30T06:38:27.403ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2785ad0c-7dc5-4c5a-9f5a-af6f186e5993", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64082025-04-30T06:38:27.406ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64092025-04-30T06:38:27.406ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64102025-04-30T06:38:27.406ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
64112025-04-30T06:38:27.408ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64122025-04-30T06:38:27.408ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64132025-04-30T06:38:27.408ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64142025-04-30T06:38:27.408ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64152025-04-30T06:38:27.409ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64162025-04-30T06:38:27.409ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64172025-04-30T06:38:27.423ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64182025-04-30T06:38:27.423ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64192025-04-30T06:38:27.423ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
64202025-04-30T06:38:27.424ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64212025-04-30T06:38:27.428ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64222025-04-30T06:38:27.438ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64232025-04-30T06:38:27.439ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64242025-04-30T06:38:27.439ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64252025-04-30T06:38:27.439ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64262025-04-30T06:38:27.444ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64272025-04-30T06:38:27.444ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
64282025-04-30T06:38:27.444ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64292025-04-30T06:38:27.470ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
64302025-04-30T06:38:27.470ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
64312025-04-30T06:38:27.470ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 408.579167ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
64322025-04-30T06:38:27.471ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
64332025-04-30T06:38:27.471ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64342025-04-30T06:38:27.471ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64352025-04-30T06:38:27.471ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64362025-04-30T06:38:27.471ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64372025-04-30T06:38:27.472ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64382025-04-30T06:38:27.475ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64392025-04-30T06:38:27.481ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64402025-04-30T06:38:27.481ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64412025-04-30T06:38:27.481ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64422025-04-30T06:38:27.482ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64432025-04-30T06:38:27.489ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
64442025-04-30T06:38:27.489ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
64452025-04-30T06:38:27.490ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
64462025-04-30T06:38:27.491ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 90 fe 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55460 req_id = f13f1ab5-d998-4fcb-8d9a-a5344174b3eb
64472025-04-30T06:38:27.491ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37118,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55460 req_id = f13f1ab5-d998-4fcb-8d9a-a5344174b3eb
64482025-04-30T06:38:27.492ZDEBGtest_quotas (store): query key key = _mgs._tcp
64492025-04-30T06:38:27.492ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64502025-04-30T06:38:27.492ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55460 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f13f1ab5-d998-4fcb-8d9a-a5344174b3eb
64512025-04-30T06:38:27.493ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3972, tv_nsec: 292055366 } })
64522025-04-30T06:38:27.493ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
64532025-04-30T06:38:27.493ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f2 c8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53418 req_id = 7ea195c7-e5cf-4605-a2fa-4f680a31465e
64542025-04-30T06:38:27.493ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62152,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53418 req_id = 7ea195c7-e5cf-4605-a2fa-4f680a31465e
64552025-04-30T06:38:27.493ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64562025-04-30T06:38:27.493ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53418 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7ea195c7-e5cf-4605-a2fa-4f680a31465e
64572025-04-30T06:38:27.557ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
64582025-04-30T06:38:27.557ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64592025-04-30T06:38:27.557ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64602025-04-30T06:38:27.557ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64612025-04-30T06:38:27.557ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
64622025-04-30T06:38:27.558ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64632025-04-30T06:38:27.559ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: cf 1f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49273 req_id = f456e7df-5553-47dc-abef-060709c730ab
64642025-04-30T06:38:27.559ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53023,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49273 req_id = f456e7df-5553-47dc-abef-060709c730ab
64652025-04-30T06:38:27.559ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
64662025-04-30T06:38:27.560ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53023, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49273 req_id = f456e7df-5553-47dc-abef-060709c730ab
64672025-04-30T06:38:27.560ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
64682025-04-30T06:38:27.560ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
64692025-04-30T06:38:27.560ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
64702025-04-30T06:38:27.561ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64712025-04-30T06:38:27.568ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64722025-04-30T06:38:27.568ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64732025-04-30T06:38:27.568ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
64742025-04-30T06:38:27.568ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64752025-04-30T06:38:27.570ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8ac96a9-e908-463f-8f5b-adf14d44ea35", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64762025-04-30T06:38:27.570ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
64772025-04-30T06:38:27.572ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64782025-04-30T06:38:27.572ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
64792025-04-30T06:38:27.572ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64802025-04-30T06:38:27.578ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fe28fc4-4fd6-4181-89fd-58d19a29cee5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64812025-04-30T06:38:27.578ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
64822025-04-30T06:38:27.582ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9410fa23-63f1-47e7-b4e7-149f24a244b0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64832025-04-30T06:38:27.582ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
64842025-04-30T06:38:27.585ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19fe41c9-c227-4b43-8ad2-51e534a038b9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64852025-04-30T06:38:27.585ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
64862025-04-30T06:38:27.587ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "454a96bd-217f-4605-a166-d8ff9fe3150f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64872025-04-30T06:38:27.588ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
64882025-04-30T06:38:27.591ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2c6e39d-f524-4dfa-85e6-b872f1347a0e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64892025-04-30T06:38:27.591ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
64902025-04-30T06:38:27.596ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fd69d74-5d7e-48b9-8046-8db51ad34ee7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64912025-04-30T06:38:27.596ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
64922025-04-30T06:38:27.601ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1a29529-45bd-4e25-8045-5123688dfb9b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64932025-04-30T06:38:27.601ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
64942025-04-30T06:38:27.604ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72d084b9-96e8-400c-a7d6-6042a75c26c4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64952025-04-30T06:38:27.604ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
64962025-04-30T06:38:27.605ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
64972025-04-30T06:38:27.606ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "987f7b2e-95f3-4feb-b282-fafbc67af8de", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
64982025-04-30T06:38:27.606ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
64992025-04-30T06:38:27.609ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3ff5469-af79-4111-9dba-29f31de11771", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65002025-04-30T06:38:27.609ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
65012025-04-30T06:38:27.610ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3aa46b5-851a-4dad-8cc0-4d4519afefce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65022025-04-30T06:38:27.610ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
65032025-04-30T06:38:27.616ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74880fa9-2376-478b-94bc-f1e38305d734", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65042025-04-30T06:38:27.616ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
65052025-04-30T06:38:27.618ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6020565-7c09-4022-a53a-6fd0366bb3fa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65062025-04-30T06:38:27.618ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
65072025-04-30T06:38:27.620ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95d90549-7f33-4382-a2a4-a801ca821424", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65082025-04-30T06:38:27.620ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
65092025-04-30T06:38:27.632ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b52c2c1-a90b-4780-b732-421cc678b341", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65102025-04-30T06:38:27.632ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
65112025-04-30T06:38:27.635ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05fbf8f1-4585-4b88-8763-151775e607e0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65122025-04-30T06:38:27.635ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
65132025-04-30T06:38:27.639ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6de1a0f8-a570-4c06-a3bb-79a8b370e3f7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65142025-04-30T06:38:27.639ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
65152025-04-30T06:38:27.643ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7299eae8-62eb-459e-9c76-433bd0537fa9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65162025-04-30T06:38:27.644ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
65172025-04-30T06:38:27.646ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fca87f20-e1ac-4a8d-8b3e-430f1fd5257d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65182025-04-30T06:38:27.646ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
65192025-04-30T06:38:27.646ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44567fb1-e0ba-4d8f-9c69-aba756da1f75", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65202025-04-30T06:38:27.646ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
65212025-04-30T06:38:27.649ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d81ff26d-8856-429c-8764-f2f6cee64b2f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65222025-04-30T06:38:27.649ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
65232025-04-30T06:38:27.650ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f221f42-ab48-43fc-bf88-74875ce109e7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65242025-04-30T06:38:27.650ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
65252025-04-30T06:38:27.651ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7c7725d-cc94-4051-a598-2d5e72adb302", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65262025-04-30T06:38:27.651ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
65272025-04-30T06:38:27.655ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74d05549-331e-4242-b2ea-36e37181ff10", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65282025-04-30T06:38:27.655ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
65292025-04-30T06:38:27.656ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65302025-04-30T06:38:27.656ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65312025-04-30T06:38:27.656ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65322025-04-30T06:38:27.656ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
65332025-04-30T06:38:27.656ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65342025-04-30T06:38:27.661ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65352025-04-30T06:38:27.668ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65362025-04-30T06:38:27.668ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65372025-04-30T06:38:27.668ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
65382025-04-30T06:38:27.669ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65392025-04-30T06:38:27.670ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d0e61b9-3b95-4972-af10-b82e7bad13b2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65402025-04-30T06:38:27.671ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
65412025-04-30T06:38:27.673ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65422025-04-30T06:38:27.674ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
65432025-04-30T06:38:27.674ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65442025-04-30T06:38:27.676ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36344d86-7c86-4704-b8b8-148cf59b6d8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65452025-04-30T06:38:27.676ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
65462025-04-30T06:38:27.684ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6e4d346-b8b7-45b4-9943-063ee638efde", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65472025-04-30T06:38:27.684ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
65482025-04-30T06:38:27.686ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4786439c-ab41-4d43-b96f-46b0763c9be9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65492025-04-30T06:38:27.686ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
65502025-04-30T06:38:27.689ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1440815-c04a-422b-8865-6b8ac8fd96e6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65512025-04-30T06:38:27.689ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
65522025-04-30T06:38:27.690ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ac43dd2-c4eb-4b43-a642-af81f3091daf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65532025-04-30T06:38:27.690ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
65542025-04-30T06:38:27.691ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2be4831f-a6bc-46a2-9b25-3bc6e8164bd0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65552025-04-30T06:38:27.691ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
65562025-04-30T06:38:27.692ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf4014af-9bc5-4c02-891a-459bbb31f06b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65572025-04-30T06:38:27.692ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
65582025-04-30T06:38:27.693ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84a7c6b0-1bf2-41b9-b143-18e280e944c8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65592025-04-30T06:38:27.693ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
65602025-04-30T06:38:27.695ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a02976c-aa26-45ec-be11-c6ef9be1a247", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65612025-04-30T06:38:27.695ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
65622025-04-30T06:38:27.697ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff7054e7-ffa7-4646-82fd-b6bc2f7c93f6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65632025-04-30T06:38:27.697ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
65642025-04-30T06:38:27.699ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c53db52d-f326-4f30-ba24-d8f79f6827b3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65652025-04-30T06:38:27.699ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
65662025-04-30T06:38:27.701ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77a41cac-114c-458b-9a86-ff24f7fab8ad", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65672025-04-30T06:38:27.701ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
65682025-04-30T06:38:27.703ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cf73c65-159c-4a05-b35f-c99cfb2ba38d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65692025-04-30T06:38:27.703ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
65702025-04-30T06:38:27.708ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
65712025-04-30T06:38:27.716ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d21080e-661b-4503-a017-f2875c82104e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65722025-04-30T06:38:27.716ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
65732025-04-30T06:38:27.718ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62cbed26-1e4d-4587-8753-056e275b3fd5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65742025-04-30T06:38:27.719ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
65752025-04-30T06:38:27.719ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5c8d0d8-3745-4724-8692-e731ab23774b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65762025-04-30T06:38:27.719ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
65772025-04-30T06:38:27.720ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90b7581b-f758-4d5f-9132-62983667f504", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65782025-04-30T06:38:27.720ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
65792025-04-30T06:38:27.721ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bf0d1a0-fbba-4936-9961-c2de488fcb03", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65802025-04-30T06:38:27.722ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
65812025-04-30T06:38:27.723ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c87ab398-24d4-4d13-b4df-77d13db3227e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
65822025-04-30T06:38:27.723ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
65832025-04-30T06:38:27.723ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65842025-04-30T06:38:27.723ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65852025-04-30T06:38:27.723ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65862025-04-30T06:38:27.724ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65872025-04-30T06:38:27.726ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65882025-04-30T06:38:27.731ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65892025-04-30T06:38:27.731ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
65902025-04-30T06:38:27.731ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65912025-04-30T06:38:27.731ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65922025-04-30T06:38:27.731ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65932025-04-30T06:38:27.732ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65942025-04-30T06:38:27.735ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65952025-04-30T06:38:27.739ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65962025-04-30T06:38:27.739ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65972025-04-30T06:38:27.739ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65982025-04-30T06:38:27.740ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65992025-04-30T06:38:27.740ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
66002025-04-30T06:38:27.741ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
66012025-04-30T06:38:27.742ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 2886ed60-d49d-4a72-93f1-cb733d03ceae uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
66022025-04-30T06:38:27.745ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 2886ed60-d49d-4a72-93f1-cb733d03ceae resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
66032025-04-30T06:38:27.745ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 2886ed60-d49d-4a72-93f1-cb733d03ceae roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
66042025-04-30T06:38:27.745ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 2886ed60-d49d-4a72-93f1-cb733d03ceae resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
66052025-04-30T06:38:27.817ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
66062025-04-30T06:38:27.817ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
66072025-04-30T06:38:27.818ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66082025-04-30T06:38:27.818ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66092025-04-30T06:38:27.818ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66102025-04-30T06:38:27.818ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
66112025-04-30T06:38:27.820ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66122025-04-30T06:38:27.820ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:38949 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
66132025-04-30T06:38:27.821ZINFOtest_quotas (dropshot_internal): request completed latency_us = 79239 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 2886ed60-d49d-4a72-93f1-cb733d03ceae response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
66142025-04-30T06:38:27.823ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:38949 req_id = f76050ff-d987-45b4-90ba-e749aa27643e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66152025-04-30T06:38:27.824ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "2886ed60-d49d-4a72-93f1-cb733d03ceae", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
66162025-04-30T06:38:27.824ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
66172025-04-30T06:38:27.825ZINFOtest_quotas (dropshot): request completed latency_us = 2272 local_addr = [::1]:55655 method = GET remote_addr = [::1]:38949 req_id = f76050ff-d987-45b4-90ba-e749aa27643e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66182025-04-30T06:38:27.845ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66192025-04-30T06:38:27.853ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66202025-04-30T06:38:27.853ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66212025-04-30T06:38:27.853ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
66222025-04-30T06:38:27.853ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66232025-04-30T06:38:27.854ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1fce9356-1f7e-4718-961b-d4077050e918 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
66242025-04-30T06:38:27.854ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1fce9356-1f7e-4718-961b-d4077050e918 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
66252025-04-30T06:38:27.854ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1fce9356-1f7e-4718-961b-d4077050e918 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
66262025-04-30T06:38:27.855ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1fce9356-1f7e-4718-961b-d4077050e918 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
66272025-04-30T06:38:27.855ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f76050ff-d987-45b4-90ba-e749aa27643e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
66282025-04-30T06:38:27.857ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66292025-04-30T06:38:27.857ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66302025-04-30T06:38:27.857ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
66312025-04-30T06:38:27.860ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66322025-04-30T06:38:27.860ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66332025-04-30T06:38:27.860ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66342025-04-30T06:38:27.860ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66352025-04-30T06:38:27.860ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66362025-04-30T06:38:27.861ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66372025-04-30T06:38:27.861ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66382025-04-30T06:38:27.865ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66392025-04-30T06:38:27.865ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
66402025-04-30T06:38:27.866ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66412025-04-30T06:38:27.866ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66422025-04-30T06:38:27.868ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66432025-04-30T06:38:27.868ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66442025-04-30T06:38:27.868ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66452025-04-30T06:38:27.870ZINFOtest_quotas (dropshot_internal): request completed latency_us = 16142 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1fce9356-1f7e-4718-961b-d4077050e918 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
66462025-04-30T06:38:27.870ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66472025-04-30T06:38:27.875ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66482025-04-30T06:38:27.876ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66492025-04-30T06:38:27.876ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66502025-04-30T06:38:27.877ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66512025-04-30T06:38:27.878ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fce9356-1f7e-4718-961b-d4077050e918", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
66522025-04-30T06:38:27.878ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
66532025-04-30T06:38:27.879ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66542025-04-30T06:38:27.879ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
66552025-04-30T06:38:27.992ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
66562025-04-30T06:38:28.008ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
66572025-04-30T06:38:28.009ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
66582025-04-30T06:38:28.009ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 538.500908ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
66592025-04-30T06:38:28.009ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
66602025-04-30T06:38:28.009ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66612025-04-30T06:38:28.009ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66622025-04-30T06:38:28.009ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66632025-04-30T06:38:28.010ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66642025-04-30T06:38:28.010ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66652025-04-30T06:38:28.016ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66662025-04-30T06:38:28.021ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66672025-04-30T06:38:28.021ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66682025-04-30T06:38:28.021ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66692025-04-30T06:38:28.022ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66702025-04-30T06:38:28.073ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66712025-04-30T06:38:28.073ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66722025-04-30T06:38:28.073ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66732025-04-30T06:38:28.073ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
66742025-04-30T06:38:28.075ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66752025-04-30T06:38:28.075ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
66762025-04-30T06:38:28.075ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
66772025-04-30T06:38:28.077ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e7 7b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .{..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55069 req_id = da714da1-fb0e-4607-9538-60491d58e450
66782025-04-30T06:38:28.078ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59259,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55069 req_id = da714da1-fb0e-4607-9538-60491d58e450
66792025-04-30T06:38:28.078ZDEBGtest_quotas (store): query key key = _mgs._tcp
66802025-04-30T06:38:28.079ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66812025-04-30T06:38:28.079ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55069 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = da714da1-fb0e-4607-9538-60491d58e450
66822025-04-30T06:38:28.079ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66832025-04-30T06:38:28.098ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66842025-04-30T06:38:28.098ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66852025-04-30T06:38:28.098ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
66862025-04-30T06:38:28.099ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66872025-04-30T06:38:28.099ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3972, tv_nsec: 898525528 } })
66882025-04-30T06:38:28.099ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
66892025-04-30T06:38:28.103ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66902025-04-30T06:38:28.103ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
66912025-04-30T06:38:28.104ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66922025-04-30T06:38:28.104ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bd 51 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .Q..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63710 req_id = f666a2f5-8ac3-41c6-ae96-b317a879ab57
66932025-04-30T06:38:28.104ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48465,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63710 req_id = f666a2f5-8ac3-41c6-ae96-b317a879ab57
66942025-04-30T06:38:28.104ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66952025-04-30T06:38:28.104ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63710 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f666a2f5-8ac3-41c6-ae96-b317a879ab57
66962025-04-30T06:38:28.163ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
66972025-04-30T06:38:28.164ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
66982025-04-30T06:38:28.164ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
66992025-04-30T06:38:28.164ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
67002025-04-30T06:38:28.165ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
67012025-04-30T06:38:28.170ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
67022025-04-30T06:38:28.170ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
67032025-04-30T06:38:28.180ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7b c0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c {............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56946 req_id = 330186ae-15fe-44fa-9cd9-d298cb0ef76d
67042025-04-30T06:38:28.180ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31680,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56946 req_id = 330186ae-15fe-44fa-9cd9-d298cb0ef76d
67052025-04-30T06:38:28.180ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
67062025-04-30T06:38:28.180ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31680, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56946 req_id = 330186ae-15fe-44fa-9cd9-d298cb0ef76d
67072025-04-30T06:38:28.180ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
67082025-04-30T06:38:28.180ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
67092025-04-30T06:38:28.180ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
67102025-04-30T06:38:28.180ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
67112025-04-30T06:38:28.181ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
67122025-04-30T06:38:28.181ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b94e4e01-78ce-446c-8286-410c51a3e599", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67132025-04-30T06:38:28.181ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
67142025-04-30T06:38:28.182ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e15c1e2-afd9-43f1-9126-2bc5b7de82ba", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67152025-04-30T06:38:28.195ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
67162025-04-30T06:38:28.202ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4b08af3-0aef-495d-a9d8-48141def8634", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67172025-04-30T06:38:28.202ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
67182025-04-30T06:38:28.204ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca844f4a-5b14-41aa-bc8d-46c84e76c09d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67192025-04-30T06:38:28.204ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
67202025-04-30T06:38:28.213ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d808dbb-39cd-4467-8f8b-4f214e8c221e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67212025-04-30T06:38:28.213ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
67222025-04-30T06:38:28.217ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da0d4680-bcbe-4129-884d-d94f6cf00174", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67232025-04-30T06:38:28.217ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
67242025-04-30T06:38:28.241ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8544fc48-01e8-4063-a8d9-33ce2d0b4554", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67252025-04-30T06:38:28.241ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
67262025-04-30T06:38:28.269ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
67272025-04-30T06:38:28.271ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47be898e-236f-474a-ad5a-f354bfe70fac", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67282025-04-30T06:38:28.271ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
67292025-04-30T06:38:28.274ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85ad093a-11ca-4234-8cf4-c01ee747e8fb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67302025-04-30T06:38:28.274ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
67312025-04-30T06:38:28.277ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
67322025-04-30T06:38:28.282ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ff0c81a-5774-44ca-824a-e92b37bf16a6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67332025-04-30T06:38:28.283ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
67342025-04-30T06:38:28.349ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67352025-04-30T06:38:28.349ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67362025-04-30T06:38:28.350ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67372025-04-30T06:38:28.350ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
67382025-04-30T06:38:28.350ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67392025-04-30T06:38:28.363ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e46941e0-f4a8-4a55-a82d-bdefac8491ab", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67402025-04-30T06:38:28.363ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
67412025-04-30T06:38:28.364ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6ce5ddc-1fb4-4205-b002-84a786e9f5a6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67422025-04-30T06:38:28.364ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
67432025-04-30T06:38:28.365ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40702185-63c8-47db-bd7c-8452ab68de6e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67442025-04-30T06:38:28.366ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
67452025-04-30T06:38:28.366ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67462025-04-30T06:38:28.372ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67472025-04-30T06:38:28.372ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67482025-04-30T06:38:28.372ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
67492025-04-30T06:38:28.373ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67502025-04-30T06:38:28.375ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d10a5ab-7f6e-439a-816f-5b786a91fd2c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:27 GMT"} })
67512025-04-30T06:38:28.375ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
67522025-04-30T06:38:28.376ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67532025-04-30T06:38:28.376ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
67542025-04-30T06:38:28.376ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67552025-04-30T06:38:28.463ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9143f67e-5e25-4fac-baf6-82e9db777ba9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67562025-04-30T06:38:28.464ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
67572025-04-30T06:38:28.470ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "affa95a3-d469-44c3-998e-9e71c6688fb8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67582025-04-30T06:38:28.470ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
67592025-04-30T06:38:28.475ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13f6ee93-2b0f-44f6-aeb8-bf9bb0547be1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67602025-04-30T06:38:28.476ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
67612025-04-30T06:38:28.481ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10fa0fb6-e3c2-4b1e-ae2b-4e801d453858", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67622025-04-30T06:38:28.481ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
67632025-04-30T06:38:28.485ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfc10c01-5e62-4151-95f8-0e16cf905838", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67642025-04-30T06:38:28.485ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
67652025-04-30T06:38:28.490ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
67662025-04-30T06:38:28.491ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9e36028-e377-4e62-9251-d023f0209948", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67672025-04-30T06:38:28.491ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
67682025-04-30T06:38:28.493ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfddc5cf-bf1c-4a40-9dcd-b9496edd3060", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67692025-04-30T06:38:28.493ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
67702025-04-30T06:38:28.495ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b674c212-3c0f-41c5-954f-9e2ed6700676", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67712025-04-30T06:38:28.495ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
67722025-04-30T06:38:28.497ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15b7c7d9-250e-429e-a594-fbe980c21dce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67732025-04-30T06:38:28.497ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
67742025-04-30T06:38:28.499ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21d025fd-33a1-4e4b-9bd0-ff8b5acdce12", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67752025-04-30T06:38:28.499ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
67762025-04-30T06:38:28.502ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47531473-355c-453c-935a-82511dbfdff9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67772025-04-30T06:38:28.503ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
67782025-04-30T06:38:28.504ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ddeab26-abba-4d78-808b-8a7b81bbee6d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67792025-04-30T06:38:28.504ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
67802025-04-30T06:38:28.506ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "776684e8-b395-49fd-8ce0-661391b7dfc3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67812025-04-30T06:38:28.506ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
67822025-04-30T06:38:28.508ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45724d8b-be0b-438c-932d-03be442505bd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67832025-04-30T06:38:28.508ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
67842025-04-30T06:38:28.509ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01ceea09-5835-485b-a51f-9f169ade5016", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67852025-04-30T06:38:28.509ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
67862025-04-30T06:38:28.511ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f61cb8a5-ebc2-4db8-b55b-5a3c94b8a7f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67872025-04-30T06:38:28.511ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
67882025-04-30T06:38:28.514ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cab08ac2-dc09-4df1-b3f4-9e128f8dd0d0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67892025-04-30T06:38:28.514ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
67902025-04-30T06:38:28.515ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9570323f-5b42-4f59-b2fe-61587050f569", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67912025-04-30T06:38:28.516ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
67922025-04-30T06:38:28.517ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25efa254-444d-4c8c-91e0-4634cee382f2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67932025-04-30T06:38:28.517ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
67942025-04-30T06:38:28.519ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e3a3f0b-4ab0-4ff8-853b-57680afc0f63", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67952025-04-30T06:38:28.519ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
67962025-04-30T06:38:28.521ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9194087e-d395-418e-af8b-10eded433757", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67972025-04-30T06:38:28.521ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
67982025-04-30T06:38:28.524ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75c08090-bf24-4cb1-86e4-1635e4c26927", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
67992025-04-30T06:38:28.524ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
68002025-04-30T06:38:28.527ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d99f21df-b771-4391-af3b-47a8420991f5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
68012025-04-30T06:38:28.527ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
68022025-04-30T06:38:28.529ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27770cc1-599f-4ff0-b34f-de341839aca0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
68032025-04-30T06:38:28.529ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
68042025-04-30T06:38:28.532ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c2dc004-670e-46d1-b275-2caea98ad14a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
68052025-04-30T06:38:28.532ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
68062025-04-30T06:38:28.534ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70dce030-f955-4f7d-8ce4-f3ffffca0295", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
68072025-04-30T06:38:28.535ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
68082025-04-30T06:38:28.537ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff2bc744-dbba-410f-8aa1-2dfbc921bbf4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
68092025-04-30T06:38:28.537ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
68102025-04-30T06:38:28.538ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2bc8fa2-4035-42aa-8482-3a812ed70fbc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
68112025-04-30T06:38:28.539ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
68122025-04-30T06:38:28.540ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89b32bd5-ceb0-4960-a49b-5e48e48c77b2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
68132025-04-30T06:38:28.541ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
68142025-04-30T06:38:28.541ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68152025-04-30T06:38:28.541ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68162025-04-30T06:38:28.541ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68172025-04-30T06:38:28.541ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
68182025-04-30T06:38:28.542ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68192025-04-30T06:38:28.581ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c098f9ae-2c8e-4aaf-9048-cf7195e53978", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
68202025-04-30T06:38:28.581ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
68212025-04-30T06:38:28.582ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68222025-04-30T06:38:28.592ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68232025-04-30T06:38:28.592ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68242025-04-30T06:38:28.592ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
68252025-04-30T06:38:28.593ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68262025-04-30T06:38:28.593ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eb1ce0f-e4a5-4ef3-b503-dc1525b688eb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
68272025-04-30T06:38:28.594ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68282025-04-30T06:38:28.594ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68292025-04-30T06:38:28.594ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68302025-04-30T06:38:28.594ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68312025-04-30T06:38:28.595ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68322025-04-30T06:38:28.598ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68332025-04-30T06:38:28.598ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
68342025-04-30T06:38:28.599ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68352025-04-30T06:38:28.602ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68362025-04-30T06:38:28.614ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68372025-04-30T06:38:28.621ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68382025-04-30T06:38:28.621ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68392025-04-30T06:38:28.621ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68402025-04-30T06:38:28.621ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68412025-04-30T06:38:28.623ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68422025-04-30T06:38:28.628ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68432025-04-30T06:38:28.635ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68442025-04-30T06:38:28.635ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68452025-04-30T06:38:28.635ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68462025-04-30T06:38:28.636ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68472025-04-30T06:38:28.715ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
68482025-04-30T06:38:28.715ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
68492025-04-30T06:38:28.715ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:60839 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
68502025-04-30T06:38:28.716ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:60839 req_id = c2553946-355f-474a-8bd0-5675d697db0a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68512025-04-30T06:38:28.717ZINFOtest_quotas (dropshot): request completed latency_us = 1053 local_addr = [::1]:55655 method = GET remote_addr = [::1]:60839 req_id = c2553946-355f-474a-8bd0-5675d697db0a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68522025-04-30T06:38:28.718ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2553946-355f-474a-8bd0-5675d697db0a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
68532025-04-30T06:38:28.720ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68542025-04-30T06:38:28.720ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68552025-04-30T06:38:28.720ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
68562025-04-30T06:38:28.721ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68572025-04-30T06:38:28.721ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68582025-04-30T06:38:28.721ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68592025-04-30T06:38:28.722ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68602025-04-30T06:38:28.723ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68612025-04-30T06:38:28.738ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68622025-04-30T06:38:28.743ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68632025-04-30T06:38:28.743ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68642025-04-30T06:38:28.743ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68652025-04-30T06:38:28.744ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68662025-04-30T06:38:28.772ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
68672025-04-30T06:38:28.841ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68682025-04-30T06:38:28.841ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68692025-04-30T06:38:28.841ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68702025-04-30T06:38:28.841ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
68712025-04-30T06:38:28.842ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68722025-04-30T06:38:28.844ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
68732025-04-30T06:38:28.844ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
68742025-04-30T06:38:28.845ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 835.402204ms iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
68752025-04-30T06:38:28.845ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
68762025-04-30T06:38:28.845ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68772025-04-30T06:38:28.845ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68782025-04-30T06:38:28.845ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68792025-04-30T06:38:28.846ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68802025-04-30T06:38:28.846ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68812025-04-30T06:38:28.859ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68822025-04-30T06:38:28.869ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68832025-04-30T06:38:28.870ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68842025-04-30T06:38:28.870ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
68852025-04-30T06:38:28.870ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68862025-04-30T06:38:28.870ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68872025-04-30T06:38:28.870ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68882025-04-30T06:38:28.871ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68892025-04-30T06:38:28.875ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68902025-04-30T06:38:28.875ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68912025-04-30T06:38:28.875ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68922025-04-30T06:38:28.875ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68932025-04-30T06:38:28.878ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68942025-04-30T06:38:28.878ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
68952025-04-30T06:38:28.878ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68962025-04-30T06:38:28.878ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68972025-04-30T06:38:28.879ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68982025-04-30T06:38:28.879ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
68992025-04-30T06:38:28.879ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
69002025-04-30T06:38:28.880ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
69012025-04-30T06:38:28.880ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
69022025-04-30T06:38:28.881ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 66 bf 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 f............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56613 req_id = c027f231-40f1-4c2c-be35-9d03f5d92861
69032025-04-30T06:38:28.882ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26303,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56613 req_id = c027f231-40f1-4c2c-be35-9d03f5d92861
69042025-04-30T06:38:28.882ZDEBGtest_quotas (store): query key key = _mgs._tcp
69052025-04-30T06:38:28.882ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
69062025-04-30T06:38:28.882ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56613 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c027f231-40f1-4c2c-be35-9d03f5d92861
69072025-04-30T06:38:28.883ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3973, tv_nsec: 682764442 } })
69082025-04-30T06:38:28.884ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
69092025-04-30T06:38:28.904ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f6 07 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61326 req_id = 6e90fbcd-35a7-49cb-a83b-1bdc8de233c0
69102025-04-30T06:38:28.927ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62983,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61326 req_id = 6e90fbcd-35a7-49cb-a83b-1bdc8de233c0
69112025-04-30T06:38:28.927ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
69122025-04-30T06:38:28.928ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61326 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6e90fbcd-35a7-49cb-a83b-1bdc8de233c0
69132025-04-30T06:38:29.030ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
69142025-04-30T06:38:29.032ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 53 d7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c S............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60940 req_id = 2ac14485-306a-47c2-b648-008d425ccc2e
69152025-04-30T06:38:29.032ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21463,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60940 req_id = 2ac14485-306a-47c2-b648-008d425ccc2e
69162025-04-30T06:38:29.032ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
69172025-04-30T06:38:29.032ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21463, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60940 req_id = 2ac14485-306a-47c2-b648-008d425ccc2e
69182025-04-30T06:38:29.034ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
69192025-04-30T06:38:29.034ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
69202025-04-30T06:38:29.034ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
69212025-04-30T06:38:29.052ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a64d8450-0ccc-4917-bc38-c2272c5fe385", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69222025-04-30T06:38:29.053ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
69232025-04-30T06:38:29.119ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69242025-04-30T06:38:29.119ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69252025-04-30T06:38:29.119ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "665d5b30-316b-4635-bc0d-bd4c766c5a98", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69262025-04-30T06:38:29.120ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
69272025-04-30T06:38:29.120ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
69282025-04-30T06:38:29.121ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
69292025-04-30T06:38:29.131ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
69302025-04-30T06:38:29.132ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
69312025-04-30T06:38:29.132ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
69322025-04-30T06:38:29.152ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c390728-0114-4d4c-9402-7cb3692e46e2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69332025-04-30T06:38:29.152ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
69342025-04-30T06:38:29.190ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69352025-04-30T06:38:29.190ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69362025-04-30T06:38:29.190ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69372025-04-30T06:38:29.190ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
69382025-04-30T06:38:29.194ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69392025-04-30T06:38:29.194ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
69402025-04-30T06:38:29.195ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
69412025-04-30T06:38:29.197ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69422025-04-30T06:38:29.210ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69432025-04-30T06:38:29.210ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69442025-04-30T06:38:29.210ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
69452025-04-30T06:38:29.213ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69462025-04-30T06:38:29.214ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ff179ad-4853-4638-a6ca-ba6c4034257b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69472025-04-30T06:38:29.214ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
69482025-04-30T06:38:29.218ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69492025-04-30T06:38:29.218ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
69502025-04-30T06:38:29.218ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69512025-04-30T06:38:29.220ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50b1cb1e-e0d8-48e4-ac65-cf2c5be153eb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69522025-04-30T06:38:29.221ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
69532025-04-30T06:38:29.221ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
69542025-04-30T06:38:29.223ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5ad1fd7-0fdf-4597-9640-e7429542bbeb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69552025-04-30T06:38:29.223ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
69562025-04-30T06:38:29.228ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "352cda0d-773e-49c7-9ae5-90dfd0bebd07", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69572025-04-30T06:38:29.228ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
69582025-04-30T06:38:29.231ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d8c0266-fbdf-40bf-b999-639e5bef4226", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69592025-04-30T06:38:29.231ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
69602025-04-30T06:38:29.231ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95c87d97-994a-406e-8ddb-7b3371708c4a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69612025-04-30T06:38:29.231ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
69622025-04-30T06:38:29.233ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c238b6f1-4fdd-4e87-a326-16d17d754f24", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69632025-04-30T06:38:29.233ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
69642025-04-30T06:38:29.234ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4386c5d9-55d6-4e71-9b0d-74e1127f9d35", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69652025-04-30T06:38:29.234ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
69662025-04-30T06:38:29.238ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05bc5bae-b331-47a8-afeb-d3abdb7f3945", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69672025-04-30T06:38:29.238ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
69682025-04-30T06:38:29.241ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bbfe49c-43a7-4d43-b846-3f4365f262cf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69692025-04-30T06:38:29.241ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
69702025-04-30T06:38:29.244ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50a45744-f9dc-4c54-bc53-d7720057a05f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69712025-04-30T06:38:29.244ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
69722025-04-30T06:38:29.248ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b70b12d1-198d-4dbb-961f-ab6fdb4b0849", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69732025-04-30T06:38:29.248ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
69742025-04-30T06:38:29.256ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7768fa7-748d-48b1-8bc4-829a7e7e3cc7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69752025-04-30T06:38:29.256ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
69762025-04-30T06:38:29.275ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e26eb717-5e50-4f81-b15e-ad2b9c1a5bd6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69772025-04-30T06:38:29.275ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
69782025-04-30T06:38:29.276ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
69792025-04-30T06:38:29.282ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfa4eef8-1128-4a15-bd13-cf238bb33a3e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69802025-04-30T06:38:29.282ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
69812025-04-30T06:38:29.286ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92f271a5-1db8-436f-bc61-029abe85e157", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69822025-04-30T06:38:29.286ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
69832025-04-30T06:38:29.288ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b7849d7-95f7-4667-bcec-95811757dd17", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69842025-04-30T06:38:29.288ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
69852025-04-30T06:38:29.290ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b933ba37-31d5-44f1-98bf-119a896a4d16", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69862025-04-30T06:38:29.290ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
69872025-04-30T06:38:29.292ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3eeca23-9c91-4a35-9074-b864abdc7f7b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69882025-04-30T06:38:29.295ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
69892025-04-30T06:38:29.309ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46a66161-e3ff-44a5-9edc-64e751c3e585", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69902025-04-30T06:38:29.309ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
69912025-04-30T06:38:29.311ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2eaab368-9c52-4eee-bf55-22b14203ce05", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69922025-04-30T06:38:29.311ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
69932025-04-30T06:38:29.315ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08e5a691-2323-4b92-b7eb-9707b5e87adf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69942025-04-30T06:38:29.315ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
69952025-04-30T06:38:29.321ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3de52fef-5adf-41d2-b7b3-2be920c22468", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
69962025-04-30T06:38:29.321ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
69972025-04-30T06:38:29.334ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69982025-04-30T06:38:29.334ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69992025-04-30T06:38:29.334ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70002025-04-30T06:38:29.334ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
70012025-04-30T06:38:29.342ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70022025-04-30T06:38:29.345ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36d79c66-cad4-42be-ba49-db9f81bef077", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
70032025-04-30T06:38:29.345ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
70042025-04-30T06:38:29.346ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70052025-04-30T06:38:29.393ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70062025-04-30T06:38:29.393ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70072025-04-30T06:38:29.393ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
70082025-04-30T06:38:29.393ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70092025-04-30T06:38:29.422ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70102025-04-30T06:38:29.422ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
70112025-04-30T06:38:29.423ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70122025-04-30T06:38:29.436ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "703d53bc-13dc-4591-8581-f900fa11229e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
70132025-04-30T06:38:29.436ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
70142025-04-30T06:38:29.441ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dec9c50-c830-45ec-9d49-cecc454b4a11", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
70152025-04-30T06:38:29.442ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
70162025-04-30T06:38:29.444ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17feeb17-879f-4039-ad26-a7854632d8ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
70172025-04-30T06:38:29.444ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
70182025-04-30T06:38:29.445ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6ffe13c-631e-4e20-83d1-05f2e8115a8f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
70192025-04-30T06:38:29.446ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
70202025-04-30T06:38:29.447ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aacf2278-8b3a-48fb-95c5-6e1053ecdaee", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:28 GMT"} })
70212025-04-30T06:38:29.455ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
70222025-04-30T06:38:29.456ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "665964fd-c0e4-4c23-bfab-46c223e0a0ad", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70232025-04-30T06:38:29.457ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
70242025-04-30T06:38:29.466ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "315e0600-ab7e-4f86-b3cb-40963b53b787", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70252025-04-30T06:38:29.467ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
70262025-04-30T06:38:29.469ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be262871-4400-4d39-aca5-ad455394ac74", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70272025-04-30T06:38:29.470ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
70282025-04-30T06:38:29.473ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8951854e-bb1d-4195-8455-8681b021bc97", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70292025-04-30T06:38:29.473ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
70302025-04-30T06:38:29.474ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
70312025-04-30T06:38:29.476ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a48b7d53-2be1-46af-8fdc-6cdf379a790e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70322025-04-30T06:38:29.477ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
70332025-04-30T06:38:29.479ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e549662-247b-46ba-9b4d-78980524f46e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70342025-04-30T06:38:29.479ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
70352025-04-30T06:38:29.482ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09bac89d-780e-4136-b741-51c956eb7146", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70362025-04-30T06:38:29.483ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
70372025-04-30T06:38:29.491ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d1f1465-cf2e-4734-8938-9988776eb8bd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70382025-04-30T06:38:29.491ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
70392025-04-30T06:38:29.491ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03d93b89-4f01-480f-bd4a-d061763c197e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70402025-04-30T06:38:29.491ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
70412025-04-30T06:38:29.491ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d388c59-42a6-4407-bf3f-bf728a558f1d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70422025-04-30T06:38:29.491ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
70432025-04-30T06:38:29.491ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cf2c9e3-d30f-4383-bd59-14842fc1b315", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70442025-04-30T06:38:29.491ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
70452025-04-30T06:38:29.491ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f23a770-7946-4057-b8f9-a2d71d402940", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70462025-04-30T06:38:29.491ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
70472025-04-30T06:38:29.492ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86786c62-2363-49d5-bef9-39d69951dcdb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70482025-04-30T06:38:29.492ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70492025-04-30T06:38:29.492ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70502025-04-30T06:38:29.492ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70512025-04-30T06:38:29.492ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70522025-04-30T06:38:29.494ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70532025-04-30T06:38:29.498ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70542025-04-30T06:38:29.504ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70552025-04-30T06:38:29.504ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70562025-04-30T06:38:29.504ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70572025-04-30T06:38:29.504ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70582025-04-30T06:38:29.504ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70592025-04-30T06:38:29.506ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70602025-04-30T06:38:29.517ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70612025-04-30T06:38:29.523ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70622025-04-30T06:38:29.523ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70632025-04-30T06:38:29.523ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70642025-04-30T06:38:29.524ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70652025-04-30T06:38:29.616ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
70662025-04-30T06:38:29.616ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
70672025-04-30T06:38:29.617ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70682025-04-30T06:38:29.617ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70692025-04-30T06:38:29.617ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70702025-04-30T06:38:29.617ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
70712025-04-30T06:38:29.618ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70722025-04-30T06:38:29.618ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:35434 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
70732025-04-30T06:38:29.619ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:35434 req_id = fd45fc8d-ecc7-41ab-883d-34b75885c825 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70742025-04-30T06:38:29.621ZINFOtest_quotas (dropshot): request completed latency_us = 1767 local_addr = [::1]:55655 method = GET remote_addr = [::1]:35434 req_id = fd45fc8d-ecc7-41ab-883d-34b75885c825 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70752025-04-30T06:38:29.622ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd45fc8d-ecc7-41ab-883d-34b75885c825", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
70762025-04-30T06:38:29.624ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70772025-04-30T06:38:29.624ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70782025-04-30T06:38:29.624ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
70792025-04-30T06:38:29.629ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70802025-04-30T06:38:29.629ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70812025-04-30T06:38:29.629ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70822025-04-30T06:38:29.629ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70832025-04-30T06:38:29.630ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70842025-04-30T06:38:29.630ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70852025-04-30T06:38:29.636ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70862025-04-30T06:38:29.636ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70872025-04-30T06:38:29.636ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
70882025-04-30T06:38:29.637ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70892025-04-30T06:38:29.654ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70902025-04-30T06:38:29.659ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70912025-04-30T06:38:29.669ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70922025-04-30T06:38:29.681ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70932025-04-30T06:38:29.682ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70942025-04-30T06:38:29.685ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70952025-04-30T06:38:29.685ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
70962025-04-30T06:38:29.685ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70972025-04-30T06:38:29.739ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
70982025-04-30T06:38:29.739ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70992025-04-30T06:38:29.740ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = abb89b51-25f0-4c2f-af73-fc3a175f361f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
71002025-04-30T06:38:29.747ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = abb89b51-25f0-4c2f-af73-fc3a175f361f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
71012025-04-30T06:38:29.747ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = abb89b51-25f0-4c2f-af73-fc3a175f361f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
71022025-04-30T06:38:29.748ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = abb89b51-25f0-4c2f-af73-fc3a175f361f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
71032025-04-30T06:38:29.763ZINFOtest_quotas (dropshot_internal): request completed latency_us = 23798 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = abb89b51-25f0-4c2f-af73-fc3a175f361f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
71042025-04-30T06:38:29.770ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "abb89b51-25f0-4c2f-af73-fc3a175f361f", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
71052025-04-30T06:38:29.771ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
71062025-04-30T06:38:29.780ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 3e17bff8-8ddc-468b-8023-18ce70b13a34 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
71072025-04-30T06:38:29.780ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 3e17bff8-8ddc-468b-8023-18ce70b13a34 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
71082025-04-30T06:38:29.780ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 3e17bff8-8ddc-468b-8023-18ce70b13a34 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
71092025-04-30T06:38:29.781ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 3e17bff8-8ddc-468b-8023-18ce70b13a34 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
71102025-04-30T06:38:29.784ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
71112025-04-30T06:38:29.784ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
71122025-04-30T06:38:29.786ZINFOtest_quotas (dropshot_internal): request completed latency_us = 5998 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 3e17bff8-8ddc-468b-8023-18ce70b13a34 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
71132025-04-30T06:38:29.798ZINFOtest_quotas (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:48722 remote_addr = 127.0.0.1:61278
71142025-04-30T06:38:29.798ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e17bff8-8ddc-468b-8023-18ce70b13a34", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
71152025-04-30T06:38:29.799ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
71162025-04-30T06:38:29.799ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71172025-04-30T06:38:29.799ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
71182025-04-30T06:38:29.800ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
71192025-04-30T06:38:29.801ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
71202025-04-30T06:38:29.801ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 956.170217ms iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
71212025-04-30T06:38:29.801ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
71222025-04-30T06:38:29.801ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
71232025-04-30T06:38:29.801ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71242025-04-30T06:38:29.801ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71252025-04-30T06:38:29.801ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71262025-04-30T06:38:29.802ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71272025-04-30T06:38:29.803ZTRACtest_quotas (dropshot): incoming request local_addr = 127.0.0.1:48722 method = GET remote_addr = 127.0.0.1:61278 req_id = 7f740ca8-bdee-46c8-8b17-1795f5538041 uri = /4678dd50-d1e0-4228-8533-c4090342ff58
71282025-04-30T06:38:29.809ZINFOtest_quotas (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 5826 local_addr = 127.0.0.1:48722 method = GET remote_addr = 127.0.0.1:61278 req_id = 7f740ca8-bdee-46c8-8b17-1795f5538041 response_code = 200 uri = /4678dd50-d1e0-4228-8533-c4090342ff58
71292025-04-30T06:38:29.811ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
71302025-04-30T06:38:29.822ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
71312025-04-30T06:38:29.822ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
71322025-04-30T06:38:29.822ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
71332025-04-30T06:38:29.822ZDEBGtest_quotas (results-sink): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71342025-04-30T06:38:29.822ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71352025-04-30T06:38:29.829ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
71362025-04-30T06:38:29.829ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71372025-04-30T06:38:29.829ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71382025-04-30T06:38:29.829ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71392025-04-30T06:38:29.829ZDEBGtest_quotas (results-sink-cluster): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71402025-04-30T06:38:29.836ZDEBGtest_quotas (clickhouse-client): unrolling 17 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
71412025-04-30T06:38:29.837ZDEBGtest_quotas (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
71422025-04-30T06:38:29.837ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
71432025-04-30T06:38:29.851ZTRACtest_quotas (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
71442025-04-30T06:38:29.851ZDEBGtest_quotas (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:38:29.837537537Z } timeseries_name = http_service:request_latency_histogram
71452025-04-30T06:38:29.856ZDEBGtest_quotas (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
71462025-04-30T06:38:29.856ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
71472025-04-30T06:38:29.856ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
71482025-04-30T06:38:29.856ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
71492025-04-30T06:38:29.857ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
71502025-04-30T06:38:29.857ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71512025-04-30T06:38:29.858ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71522025-04-30T06:38:29.860ZTRACtest_quotas (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
71532025-04-30T06:38:29.860ZDEBGtest_quotas (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:38:29.856381573Z } timeseries_name = collection_target:cpus_provisioned
71542025-04-30T06:38:29.860ZDEBGtest_quotas (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
71552025-04-30T06:38:29.860ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
71562025-04-30T06:38:29.861ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fb 27 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .'..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61146 req_id = 396a01e1-6612-4fdf-8c07-665f981b1a89
71572025-04-30T06:38:29.861ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64295,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61146 req_id = 396a01e1-6612-4fdf-8c07-665f981b1a89
71582025-04-30T06:38:29.861ZDEBGtest_quotas (store): query key key = _mgs._tcp
71592025-04-30T06:38:29.861ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
71602025-04-30T06:38:29.861ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61146 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 396a01e1-6612-4fdf-8c07-665f981b1a89
71612025-04-30T06:38:29.863ZTRACtest_quotas (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
71622025-04-30T06:38:29.863ZDEBGtest_quotas (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:38:29.860738604Z } timeseries_name = collection_target:ram_provisioned
71632025-04-30T06:38:29.864ZDEBGtest_quotas (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
71642025-04-30T06:38:29.864ZTRACtest_quotas (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
71652025-04-30T06:38:29.864ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3974, tv_nsec: 663319354 } })
71662025-04-30T06:38:29.864ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
71672025-04-30T06:38:29.864ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71682025-04-30T06:38:29.869ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71692025-04-30T06:38:29.869ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71702025-04-30T06:38:29.869ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71712025-04-30T06:38:29.870ZTRACtest_quotas (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
71722025-04-30T06:38:29.870ZDEBGtest_quotas (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:38:29.863952938Z } timeseries_name = collection_target:virtual_disk_space_provisioned
71732025-04-30T06:38:29.876ZDEBGtest_quotas (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
71742025-04-30T06:38:29.876ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
71752025-04-30T06:38:29.876ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2e a9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61055 req_id = a0f489b7-b8e9-4a95-bbad-7fd64ab61f4c
71762025-04-30T06:38:29.876ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11945,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61055 req_id = a0f489b7-b8e9-4a95-bbad-7fd64ab61f4c
71772025-04-30T06:38:29.877ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
71782025-04-30T06:38:29.877ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61055 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a0f489b7-b8e9-4a95-bbad-7fd64ab61f4c
71792025-04-30T06:38:29.924ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
71802025-04-30T06:38:29.925ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71812025-04-30T06:38:29.925ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71822025-04-30T06:38:29.925ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71832025-04-30T06:38:29.925ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
71842025-04-30T06:38:29.926ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71852025-04-30T06:38:29.926ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_string FORMAT Native
71862025-04-30T06:38:29.928ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3f 08 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ?............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54985 req_id = 856d92bf-12de-461b-9928-1296b0b3a67c
71872025-04-30T06:38:29.928ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16136,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54985 req_id = 856d92bf-12de-461b-9928-1296b0b3a67c
71882025-04-30T06:38:29.928ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
71892025-04-30T06:38:29.928ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16136, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54985 req_id = 856d92bf-12de-461b-9928-1296b0b3a67c
71902025-04-30T06:38:29.929ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
71912025-04-30T06:38:29.929ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
71922025-04-30T06:38:29.929ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
71932025-04-30T06:38:29.929ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71942025-04-30T06:38:29.934ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71952025-04-30T06:38:29.934ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71962025-04-30T06:38:29.934ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
71972025-04-30T06:38:29.935ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71982025-04-30T06:38:29.936ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 10 table_name = fields_string
71992025-04-30T06:38:29.936ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
72002025-04-30T06:38:29.939ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
72012025-04-30T06:38:29.945ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72022025-04-30T06:38:29.945ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
72032025-04-30T06:38:29.945ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72042025-04-30T06:38:29.946ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 5 table_name = fields_u16
72052025-04-30T06:38:29.946ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 17 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
72062025-04-30T06:38:29.953ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 17 table_name = fields_uuid
72072025-04-30T06:38:29.953ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
72082025-04-30T06:38:29.954ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e6c5bfe-560a-4bc7-a7cb-a11ae4e6c1e1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72092025-04-30T06:38:29.954ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
72102025-04-30T06:38:29.965ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 5 table_name = measurements_histogramu64
72112025-04-30T06:38:29.965ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
72122025-04-30T06:38:29.966ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da7b2373-b0e4-4dcb-8661-f6a766894f0d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72132025-04-30T06:38:29.966ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
72142025-04-30T06:38:29.974ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 12 table_name = measurements_i64
72152025-04-30T06:38:29.974ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
72162025-04-30T06:38:29.998ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e857190e-3bc9-4bbb-81ae-bd9c45ebc246", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72172025-04-30T06:38:29.999ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
72182025-04-30T06:38:30.006ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7a0f723-bdba-451e-bb7d-cc02e359965d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72192025-04-30T06:38:30.006ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
72202025-04-30T06:38:30.018ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32ff8811-36a4-410e-80ea-624a901790b2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72212025-04-30T06:38:30.019ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
72222025-04-30T06:38:30.020ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
72232025-04-30T06:38:30.022ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f208b6f4-7df8-43ef-bf27-08e2a209e912", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72242025-04-30T06:38:30.022ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
72252025-04-30T06:38:30.025ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39e08de3-bf85-4a0e-8cd3-acf2a08446db", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72262025-04-30T06:38:30.025ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
72272025-04-30T06:38:30.027ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e67d28f0-93d3-4949-ae22-58e9d083cf1c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72282025-04-30T06:38:30.027ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
72292025-04-30T06:38:30.028ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "343e9570-53c7-4769-9807-9ff58a64446d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72302025-04-30T06:38:30.029ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
72312025-04-30T06:38:30.030ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3d2b045-7856-4d72-a333-4b7b9587bff4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72322025-04-30T06:38:30.030ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
72332025-04-30T06:38:30.031ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1d85652-543e-4bdb-bfe4-7288630929e1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72342025-04-30T06:38:30.031ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
72352025-04-30T06:38:30.032ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c2d5b02-a38b-49ad-a4aa-2292928f8d4d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72362025-04-30T06:38:30.032ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
72372025-04-30T06:38:30.034ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c90a27a6-ac16-4cf4-bbfa-1e51ac916fe9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72382025-04-30T06:38:30.034ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
72392025-04-30T06:38:30.037ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "214d5327-9544-4a87-994f-c22722372fe3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72402025-04-30T06:38:30.038ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
72412025-04-30T06:38:30.039ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8835c77a-8632-47a7-861c-e2a5f2de377f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72422025-04-30T06:38:30.040ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
72432025-04-30T06:38:30.044ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30a23f2c-5879-4d96-b9f6-8ad8b231d196", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72442025-04-30T06:38:30.044ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
72452025-04-30T06:38:30.050ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b21d183f-b880-4dff-9153-67f98dc3dc67", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72462025-04-30T06:38:30.050ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
72472025-04-30T06:38:30.054ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "101a9875-1c7f-4757-be33-9086348b04f2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72482025-04-30T06:38:30.054ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
72492025-04-30T06:38:30.056ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5d60349-716c-41da-b6c5-21f167348c27", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72502025-04-30T06:38:30.056ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
72512025-04-30T06:38:30.057ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca680cf3-18e7-452e-b636-bf1f06f1ea06", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72522025-04-30T06:38:30.057ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
72532025-04-30T06:38:30.060ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "901c8ff6-3d78-47ff-9ee5-4617341f5c7e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72542025-04-30T06:38:30.062ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
72552025-04-30T06:38:30.064ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59060629-2734-48d0-920e-9e1c77cb86e7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72562025-04-30T06:38:30.064ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
72572025-04-30T06:38:30.065ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeb96409-f09e-4f45-86f5-acba39c51ab0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72582025-04-30T06:38:30.065ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
72592025-04-30T06:38:30.067ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bf1b909-df85-444d-998d-1468920269a2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72602025-04-30T06:38:30.067ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
72612025-04-30T06:38:30.070ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "105a7a4e-c703-49e1-99b7-fc303fc12ae4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72622025-04-30T06:38:30.070ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
72632025-04-30T06:38:30.070ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c46062a-5892-4cbb-b8cf-b8d2a3c65bec", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72642025-04-30T06:38:30.070ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
72652025-04-30T06:38:30.071ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72662025-04-30T06:38:30.071ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72672025-04-30T06:38:30.071ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72682025-04-30T06:38:30.071ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
72692025-04-30T06:38:30.072ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72702025-04-30T06:38:30.074ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22f8d73e-cd36-4e8b-8e80-6a4a8a7628f0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72712025-04-30T06:38:30.074ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
72722025-04-30T06:38:30.074ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72732025-04-30T06:38:30.081ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72742025-04-30T06:38:30.081ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72752025-04-30T06:38:30.082ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
72762025-04-30T06:38:30.082ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72772025-04-30T06:38:30.084ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c63a7c54-3b31-4b42-8a0a-34d6c585f4be", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72782025-04-30T06:38:30.084ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
72792025-04-30T06:38:30.085ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72802025-04-30T06:38:30.085ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
72812025-04-30T06:38:30.085ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72822025-04-30T06:38:30.088ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "459739cd-3b6e-4ba1-804a-e37fbd054a22", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72832025-04-30T06:38:30.088ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
72842025-04-30T06:38:30.091ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "633cb710-f30c-4a3d-8569-f521f1a4c5ed", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72852025-04-30T06:38:30.091ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
72862025-04-30T06:38:30.092ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9f0bac6-ea38-4af1-84cf-f89e7057d203", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72872025-04-30T06:38:30.092ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
72882025-04-30T06:38:30.093ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c6f646f-2a35-436f-ab10-92858a96d4ff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72892025-04-30T06:38:30.094ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
72902025-04-30T06:38:30.095ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05354ce4-3df8-46e2-8e51-0fa74a529446", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72912025-04-30T06:38:30.095ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
72922025-04-30T06:38:30.097ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5207a900-a798-4e42-8f8d-505ef8987520", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72932025-04-30T06:38:30.097ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
72942025-04-30T06:38:30.099ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25c5cdc2-44ca-4788-865a-162b959f1ac1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72952025-04-30T06:38:30.099ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
72962025-04-30T06:38:30.101ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80de2ffa-ce89-4ced-81fe-2881f3da0b46", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
72972025-04-30T06:38:30.101ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
72982025-04-30T06:38:30.103ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
72992025-04-30T06:38:30.103ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73002025-04-30T06:38:30.103ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
73012025-04-30T06:38:30.104ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
73022025-04-30T06:38:30.105ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f17eb59-2680-4364-a8fc-fae6c0f68a82", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
73032025-04-30T06:38:30.105ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
73042025-04-30T06:38:30.108ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
73052025-04-30T06:38:30.108ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
73062025-04-30T06:38:30.108ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac2f09bb-f5a9-47c9-a42e-af810d68f42e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
73072025-04-30T06:38:30.108ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
73082025-04-30T06:38:30.111ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
73092025-04-30T06:38:30.111ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73102025-04-30T06:38:30.114ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75cab2af-9938-48f2-8f6a-9a6001d6b57b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
73112025-04-30T06:38:30.114ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
73122025-04-30T06:38:30.114ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
73132025-04-30T06:38:30.117ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33f737be-902b-462e-9a07-16fd5b8abd16", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
73142025-04-30T06:38:30.117ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
73152025-04-30T06:38:30.119ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c5555b7-6099-4b6b-b6c8-18afe1e3873d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
73162025-04-30T06:38:30.119ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
73172025-04-30T06:38:30.120ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "165b8b80-ca00-4adc-a230-e24f8c10ef9f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
73182025-04-30T06:38:30.120ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
73192025-04-30T06:38:30.121ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fbd3ed2-c6b4-4f7d-9bac-5c053ce44b12", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
73202025-04-30T06:38:30.121ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
73212025-04-30T06:38:30.122ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40e54fcf-be22-4c80-9691-3eab95255931", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
73222025-04-30T06:38:30.122ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
73232025-04-30T06:38:30.123ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa7f4665-a73a-4bb4-a44b-d1901d043f7c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
73242025-04-30T06:38:30.123ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
73252025-04-30T06:38:30.124ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73262025-04-30T06:38:30.124ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73272025-04-30T06:38:30.124ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73282025-04-30T06:38:30.124ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73292025-04-30T06:38:30.129ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73302025-04-30T06:38:30.134ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
73312025-04-30T06:38:30.134ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
73322025-04-30T06:38:30.135ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73332025-04-30T06:38:30.135ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73342025-04-30T06:38:30.135ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73352025-04-30T06:38:30.136ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73362025-04-30T06:38:30.140ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73372025-04-30T06:38:30.145ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
73382025-04-30T06:38:30.145ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73392025-04-30T06:38:30.145ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73402025-04-30T06:38:30.146ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73412025-04-30T06:38:30.210ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
73422025-04-30T06:38:30.210ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
73432025-04-30T06:38:30.211ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73442025-04-30T06:38:30.211ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73452025-04-30T06:38:30.212ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73462025-04-30T06:38:30.212ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
73472025-04-30T06:38:30.213ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73482025-04-30T06:38:30.213ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:33988 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
73492025-04-30T06:38:30.213ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
73502025-04-30T06:38:30.214ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:33988 req_id = 1eedfc0b-14f1-4ece-808d-21858f0ae7fe sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
73512025-04-30T06:38:30.214ZINFOtest_quotas (dropshot): request completed latency_us = 544 local_addr = [::1]:55655 method = GET remote_addr = [::1]:33988 req_id = 1eedfc0b-14f1-4ece-808d-21858f0ae7fe response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
73522025-04-30T06:38:30.215ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1eedfc0b-14f1-4ece-808d-21858f0ae7fe", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
73532025-04-30T06:38:30.217ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
73542025-04-30T06:38:30.218ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
73552025-04-30T06:38:30.218ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
73562025-04-30T06:38:30.220ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73572025-04-30T06:38:30.220ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73582025-04-30T06:38:30.220ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73592025-04-30T06:38:30.220ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73602025-04-30T06:38:30.220ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73612025-04-30T06:38:30.221ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73622025-04-30T06:38:30.226ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73632025-04-30T06:38:30.226ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73642025-04-30T06:38:30.227ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
73652025-04-30T06:38:30.227ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73662025-04-30T06:38:30.229ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73672025-04-30T06:38:30.234ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73682025-04-30T06:38:30.235ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73692025-04-30T06:38:30.235ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73702025-04-30T06:38:30.236ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73712025-04-30T06:38:30.239ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73722025-04-30T06:38:30.239ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
73732025-04-30T06:38:30.241ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73742025-04-30T06:38:30.265ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
73752025-04-30T06:38:30.265ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
73762025-04-30T06:38:30.265ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 463.836745ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
73772025-04-30T06:38:30.265ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
73782025-04-30T06:38:30.265ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73792025-04-30T06:38:30.265ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73802025-04-30T06:38:30.265ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73812025-04-30T06:38:30.265ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73822025-04-30T06:38:30.266ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73832025-04-30T06:38:30.266ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
73842025-04-30T06:38:30.268ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73852025-04-30T06:38:30.286ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73862025-04-30T06:38:30.286ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73872025-04-30T06:38:30.286ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73882025-04-30T06:38:30.286ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73892025-04-30T06:38:30.292ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
73902025-04-30T06:38:30.292ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
73912025-04-30T06:38:30.293ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 20 66 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 f..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52682 req_id = a82c9e65-1809-44d7-b1fb-de45de684633
73922025-04-30T06:38:30.294ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8294,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52682 req_id = a82c9e65-1809-44d7-b1fb-de45de684633
73932025-04-30T06:38:30.294ZDEBGtest_quotas (store): query key key = _mgs._tcp
73942025-04-30T06:38:30.294ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
73952025-04-30T06:38:30.294ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52682 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a82c9e65-1809-44d7-b1fb-de45de684633
73962025-04-30T06:38:30.294ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3975, tv_nsec: 93832283 } })
73972025-04-30T06:38:30.294ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
73982025-04-30T06:38:30.295ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: eb 89 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62766 req_id = fa26ca4f-c7c9-4202-983e-0ea182d45f70
73992025-04-30T06:38:30.295ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60297,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62766 req_id = fa26ca4f-c7c9-4202-983e-0ea182d45f70
74002025-04-30T06:38:30.295ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
74012025-04-30T06:38:30.295ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62766 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fa26ca4f-c7c9-4202-983e-0ea182d45f70
74022025-04-30T06:38:30.350ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
74032025-04-30T06:38:30.350ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74042025-04-30T06:38:30.350ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74052025-04-30T06:38:30.351ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74062025-04-30T06:38:30.351ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
74072025-04-30T06:38:30.351ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74082025-04-30T06:38:30.352ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 14 f1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61521 req_id = 9d631633-5651-4cca-880e-e0f4e709756f
74092025-04-30T06:38:30.352ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5361,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61521 req_id = 9d631633-5651-4cca-880e-e0f4e709756f
74102025-04-30T06:38:30.352ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
74112025-04-30T06:38:30.352ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5361, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61521 req_id = 9d631633-5651-4cca-880e-e0f4e709756f
74122025-04-30T06:38:30.353ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
74132025-04-30T06:38:30.353ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
74142025-04-30T06:38:30.353ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
74152025-04-30T06:38:30.355ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74162025-04-30T06:38:30.361ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74172025-04-30T06:38:30.361ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74182025-04-30T06:38:30.361ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
74192025-04-30T06:38:30.362ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74202025-04-30T06:38:30.365ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "13f1f2f3-3463-4149-be2f-e0f183ef0e25", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74212025-04-30T06:38:30.366ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
74222025-04-30T06:38:30.366ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74232025-04-30T06:38:30.366ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
74242025-04-30T06:38:30.366ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74252025-04-30T06:38:30.369ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7583411-d858-4681-bba5-640ac60e698f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74262025-04-30T06:38:30.369ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
74272025-04-30T06:38:30.374ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "010962af-cca6-4c68-9148-0d638e44fa8e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74282025-04-30T06:38:30.374ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
74292025-04-30T06:38:30.376ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "449bf7ac-c4fa-4dc4-b198-3237134c7ed8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74302025-04-30T06:38:30.377ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
74312025-04-30T06:38:30.380ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96f25078-3c5b-411d-b209-70a0ea3de22d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74322025-04-30T06:38:30.381ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
74332025-04-30T06:38:30.384ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93b762b8-93a2-4bb7-9287-f793a109c5ab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74342025-04-30T06:38:30.385ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
74352025-04-30T06:38:30.388ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f51305fa-b4c0-4af7-9ecd-dd1c3acc4d41", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74362025-04-30T06:38:30.388ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
74372025-04-30T06:38:30.392ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fc3c309-1581-442c-99e8-879c1b539ad0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74382025-04-30T06:38:30.392ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
74392025-04-30T06:38:30.394ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b5afe89-057a-4ae5-adc1-e36bb4ad97a7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74402025-04-30T06:38:30.395ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
74412025-04-30T06:38:30.397ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d7b52ec-6a6a-414d-8899-3fbc8f99fddc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74422025-04-30T06:38:30.397ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
74432025-04-30T06:38:30.398ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
74442025-04-30T06:38:30.399ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "536b4fc6-e435-4a66-8ddc-fd3642be7c37", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74452025-04-30T06:38:30.399ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
74462025-04-30T06:38:30.403ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44329ff4-3341-4df5-ab2d-38e5295cf1ee", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74472025-04-30T06:38:30.403ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
74482025-04-30T06:38:30.404ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37cc6627-15a7-4f70-8052-da55ff210c51", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74492025-04-30T06:38:30.405ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
74502025-04-30T06:38:30.408ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02467c57-a88c-434a-a1ab-07eaf910127d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74512025-04-30T06:38:30.408ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
74522025-04-30T06:38:30.411ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d34c44e-1408-4110-a94a-c290004171f1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74532025-04-30T06:38:30.412ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
74542025-04-30T06:38:30.414ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bb8b5d2-de6e-4990-8364-46bbe145d2e2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74552025-04-30T06:38:30.414ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
74562025-04-30T06:38:30.416ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c52137e-2dc5-4ec0-925a-a32eca481e5e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74572025-04-30T06:38:30.416ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
74582025-04-30T06:38:30.419ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfdcc01d-7396-43f5-8f5c-4a74c2e23733", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74592025-04-30T06:38:30.419ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
74602025-04-30T06:38:30.421ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5609b47-8ba9-47d4-af10-f720bb4f3ec3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74612025-04-30T06:38:30.422ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
74622025-04-30T06:38:30.423ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "198f51ab-33bf-4d7b-af21-b03836d6063d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74632025-04-30T06:38:30.424ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
74642025-04-30T06:38:30.425ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3480ef8c-6a91-4606-a4cd-1f827256810f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74652025-04-30T06:38:30.425ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
74662025-04-30T06:38:30.426ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0c862f3-2290-4403-9aa2-0f2a000c32d1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74672025-04-30T06:38:30.426ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
74682025-04-30T06:38:30.427ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d231699-e458-43be-b11e-d316da989e60", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74692025-04-30T06:38:30.428ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
74702025-04-30T06:38:30.429ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cbdb6bc-2092-4014-887e-5f2576251ec2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74712025-04-30T06:38:30.429ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
74722025-04-30T06:38:30.431ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b97d66b4-daf9-4d18-99a1-9832fa37edf6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74732025-04-30T06:38:30.431ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
74742025-04-30T06:38:30.435ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e0ec3a3-308b-4658-8dee-cee10a33c39e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74752025-04-30T06:38:30.435ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
74762025-04-30T06:38:30.437ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "787a8472-e3d0-4ebe-a2ce-44388b82416c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74772025-04-30T06:38:30.437ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
74782025-04-30T06:38:30.440ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87eda022-8ab3-4a51-868e-2a4b5f60b847", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74792025-04-30T06:38:30.441ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
74802025-04-30T06:38:30.443ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb21908e-d49c-47e0-a4de-c7981cd90d1f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74812025-04-30T06:38:30.444ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
74822025-04-30T06:38:30.446ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e0682f9-b9df-45fb-ae64-8022aaaf78e3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74832025-04-30T06:38:30.446ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
74842025-04-30T06:38:30.447ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f6fd239-f330-4e92-9eb4-9d8bdac729e8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74852025-04-30T06:38:30.447ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
74862025-04-30T06:38:30.448ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1a7585e-87b0-4558-bee9-83ae77d2168a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74872025-04-30T06:38:30.449ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
74882025-04-30T06:38:30.449ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74892025-04-30T06:38:30.449ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74902025-04-30T06:38:30.449ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74912025-04-30T06:38:30.449ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
74922025-04-30T06:38:30.450ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74932025-04-30T06:38:30.454ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44f43d02-af05-4bc0-a4bb-e2a750f5c47f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:29 GMT"} })
74942025-04-30T06:38:30.454ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
74952025-04-30T06:38:30.455ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74962025-04-30T06:38:30.461ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74972025-04-30T06:38:30.461ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74982025-04-30T06:38:30.461ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
74992025-04-30T06:38:30.462ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75002025-04-30T06:38:30.465ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "881b1fd7-81a5-4c58-bdda-43398e520a7e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75012025-04-30T06:38:30.466ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
75022025-04-30T06:38:30.468ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75032025-04-30T06:38:30.468ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
75042025-04-30T06:38:30.468ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75052025-04-30T06:38:30.471ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9123a065-456e-4c21-865e-245c1f64da50", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75062025-04-30T06:38:30.471ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
75072025-04-30T06:38:30.476ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8507beb3-f417-4519-9cf2-daeeeb39b73f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75082025-04-30T06:38:30.476ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
75092025-04-30T06:38:30.479ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfa1634c-17a1-4904-b5fd-49a3ae87ded0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75102025-04-30T06:38:30.479ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
75112025-04-30T06:38:30.483ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1b1c3a4-492e-4fe3-8959-e604e0cd0135", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75122025-04-30T06:38:30.483ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
75132025-04-30T06:38:30.487ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d4ade3d-4e66-4d57-a013-7f7a9039c75c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75142025-04-30T06:38:30.487ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
75152025-04-30T06:38:30.490ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fec00393-6a2f-473d-872a-59e665e43113", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75162025-04-30T06:38:30.490ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
75172025-04-30T06:38:30.493ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61b8fd56-7dc4-4ab0-97aa-81e9e8c17bf8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75182025-04-30T06:38:30.493ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
75192025-04-30T06:38:30.494ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "668e0b89-5f6f-4c6e-9fc3-c91bf7e3a2b7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75202025-04-30T06:38:30.495ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
75212025-04-30T06:38:30.500ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
75222025-04-30T06:38:30.501ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08bd5a57-d91b-42d4-8219-edfe184d74b2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75232025-04-30T06:38:30.501ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
75242025-04-30T06:38:30.503ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "affb6664-0fbb-4823-82c4-f35221d29a85", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75252025-04-30T06:38:30.504ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
75262025-04-30T06:38:30.505ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "426b3f82-9487-40b3-9950-b7fcc3e3cd34", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75272025-04-30T06:38:30.505ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
75282025-04-30T06:38:30.505ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75292025-04-30T06:38:30.505ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75302025-04-30T06:38:30.506ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75312025-04-30T06:38:30.506ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75322025-04-30T06:38:30.520ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75332025-04-30T06:38:30.526ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
75342025-04-30T06:38:30.526ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
75352025-04-30T06:38:30.526ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75362025-04-30T06:38:30.526ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75372025-04-30T06:38:30.526ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75382025-04-30T06:38:30.526ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75392025-04-30T06:38:30.529ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75402025-04-30T06:38:30.537ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
75412025-04-30T06:38:30.537ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75422025-04-30T06:38:30.537ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75432025-04-30T06:38:30.538ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75442025-04-30T06:38:30.636ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
75452025-04-30T06:38:30.636ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
75462025-04-30T06:38:30.637ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75472025-04-30T06:38:30.637ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75482025-04-30T06:38:30.637ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75492025-04-30T06:38:30.637ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
75502025-04-30T06:38:30.638ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75512025-04-30T06:38:30.638ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:44283 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
75522025-04-30T06:38:30.639ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:44283 req_id = c56c2761-5db1-4bf3-b0cf-412e452c5358 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
75532025-04-30T06:38:30.640ZINFOtest_quotas (dropshot): request completed latency_us = 759 local_addr = [::1]:55655 method = GET remote_addr = [::1]:44283 req_id = c56c2761-5db1-4bf3-b0cf-412e452c5358 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
75542025-04-30T06:38:30.641ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c56c2761-5db1-4bf3-b0cf-412e452c5358", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
75552025-04-30T06:38:30.643ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
75562025-04-30T06:38:30.643ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
75572025-04-30T06:38:30.643ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
75582025-04-30T06:38:30.646ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
75592025-04-30T06:38:30.646ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75602025-04-30T06:38:30.646ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75612025-04-30T06:38:30.647ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75622025-04-30T06:38:30.647ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75632025-04-30T06:38:30.648ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75642025-04-30T06:38:30.654ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75652025-04-30T06:38:30.654ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75662025-04-30T06:38:30.654ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
75672025-04-30T06:38:30.655ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75682025-04-30T06:38:30.656ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75692025-04-30T06:38:30.660ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
75702025-04-30T06:38:30.660ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75712025-04-30T06:38:30.660ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75722025-04-30T06:38:30.661ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75732025-04-30T06:38:30.661ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75742025-04-30T06:38:30.661ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
75752025-04-30T06:38:30.662ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75762025-04-30T06:38:30.702ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
75772025-04-30T06:38:30.703ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
75782025-04-30T06:38:30.703ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 438.004045ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
75792025-04-30T06:38:30.703ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
75802025-04-30T06:38:30.703ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
75812025-04-30T06:38:30.703ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75822025-04-30T06:38:30.703ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75832025-04-30T06:38:30.703ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75842025-04-30T06:38:30.705ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75852025-04-30T06:38:30.707ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75862025-04-30T06:38:30.714ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
75872025-04-30T06:38:30.715ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75882025-04-30T06:38:30.715ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75892025-04-30T06:38:30.715ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75902025-04-30T06:38:30.724ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
75912025-04-30T06:38:30.733ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
75922025-04-30T06:38:30.733ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
75932025-04-30T06:38:30.735ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e5 7d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .}..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54920 req_id = c780699b-12bd-4fa6-8339-9cc9cc6094f1
75942025-04-30T06:38:30.735ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58749,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54920 req_id = c780699b-12bd-4fa6-8339-9cc9cc6094f1
75952025-04-30T06:38:30.735ZDEBGtest_quotas (store): query key key = _mgs._tcp
75962025-04-30T06:38:30.735ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
75972025-04-30T06:38:30.735ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54920 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c780699b-12bd-4fa6-8339-9cc9cc6094f1
75982025-04-30T06:38:30.736ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3975, tv_nsec: 535021614 } })
75992025-04-30T06:38:30.736ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
76002025-04-30T06:38:30.736ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a4 dc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53867 req_id = 6f37a37b-3e25-4ac7-93b7-ab1a1c93a138
76012025-04-30T06:38:30.736ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42204,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53867 req_id = 6f37a37b-3e25-4ac7-93b7-ab1a1c93a138
76022025-04-30T06:38:30.737ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
76032025-04-30T06:38:30.737ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53867 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6f37a37b-3e25-4ac7-93b7-ab1a1c93a138
76042025-04-30T06:38:30.791ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
76052025-04-30T06:38:30.791ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76062025-04-30T06:38:30.791ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76072025-04-30T06:38:30.791ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76082025-04-30T06:38:30.791ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
76092025-04-30T06:38:30.792ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76102025-04-30T06:38:30.793ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a4 bc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60527 req_id = cc2cefc3-b11b-46e8-a769-7fddc4328e59
76112025-04-30T06:38:30.793ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42172,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60527 req_id = cc2cefc3-b11b-46e8-a769-7fddc4328e59
76122025-04-30T06:38:30.793ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
76132025-04-30T06:38:30.793ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42172, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60527 req_id = cc2cefc3-b11b-46e8-a769-7fddc4328e59
76142025-04-30T06:38:30.794ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
76152025-04-30T06:38:30.794ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
76162025-04-30T06:38:30.794ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
76172025-04-30T06:38:30.796ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76182025-04-30T06:38:30.802ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76192025-04-30T06:38:30.802ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76202025-04-30T06:38:30.803ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
76212025-04-30T06:38:30.803ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76222025-04-30T06:38:30.803ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b69fc685-066d-4471-b011-fa84650a9de8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76232025-04-30T06:38:30.804ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
76242025-04-30T06:38:30.806ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "978237b6-be54-4ff5-b7bc-09a3b35356d0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76252025-04-30T06:38:30.806ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
76262025-04-30T06:38:30.807ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76272025-04-30T06:38:30.807ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
76282025-04-30T06:38:30.807ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76292025-04-30T06:38:30.809ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7971fd51-e66d-43c6-9c2d-73b335f8aee1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76302025-04-30T06:38:30.809ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
76312025-04-30T06:38:30.812ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "11e2b642-9f2c-415a-94b5-6e46f7888067", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76322025-04-30T06:38:30.812ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
76332025-04-30T06:38:30.815ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a7d0e82-dc15-4141-b006-87f8f194e422", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76342025-04-30T06:38:30.816ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
76352025-04-30T06:38:30.820ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1538d672-250f-4328-805e-2e5ec19aab58", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76362025-04-30T06:38:30.821ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
76372025-04-30T06:38:30.828ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19fbc559-db25-4dcf-a364-2f3ad8192f4b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76382025-04-30T06:38:30.828ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
76392025-04-30T06:38:30.834ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbf66649-9bc4-4edf-a6f8-55fbdb2db286", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76402025-04-30T06:38:30.834ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
76412025-04-30T06:38:30.837ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e193091-7bc2-44ff-8191-81fe272ed25a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76422025-04-30T06:38:30.837ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
76432025-04-30T06:38:30.841ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edc79196-c89a-4326-ba07-78a7a6b603c0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76442025-04-30T06:38:30.841ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
76452025-04-30T06:38:30.841ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
76462025-04-30T06:38:30.843ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b9e9f09-00a3-4643-99cb-d67dae7d654e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76472025-04-30T06:38:30.843ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
76482025-04-30T06:38:30.845ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48547a7b-dad7-4c04-9a6f-016292df80ad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76492025-04-30T06:38:30.845ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
76502025-04-30T06:38:30.847ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d89e23d-a1ff-4eed-814b-ebaf906948fc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76512025-04-30T06:38:30.847ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
76522025-04-30T06:38:30.849ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe4e1dd7-22fa-4dda-aee4-3ea9cb035457", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76532025-04-30T06:38:30.850ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
76542025-04-30T06:38:30.852ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76552025-04-30T06:38:30.853ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76562025-04-30T06:38:30.854ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76572025-04-30T06:38:30.855ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76582025-04-30T06:38:30.855ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76592025-04-30T06:38:30.855ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76602025-04-30T06:38:30.856ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddf041cd-747f-4c8a-8a3d-ad15c688d3f1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76612025-04-30T06:38:30.856ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
76622025-04-30T06:38:30.858ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63d8437e-5f10-45be-81de-795a578b648e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76632025-04-30T06:38:30.858ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
76642025-04-30T06:38:30.860ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ff6bb51-1f25-4030-838d-2f33eb877e33", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76652025-04-30T06:38:30.860ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
76662025-04-30T06:38:30.862ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b879588-fee4-4d47-b2ff-b9d33efd55da", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76672025-04-30T06:38:30.862ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
76682025-04-30T06:38:30.864ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36b7f2d4-ca30-429e-93c0-c6598496483b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76692025-04-30T06:38:30.864ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
76702025-04-30T06:38:30.865ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8acef251-5721-44af-b599-9a0d7ec71982", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76712025-04-30T06:38:30.865ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
76722025-04-30T06:38:30.866ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b04e93a-5ced-4965-a898-b345d439ae41", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76732025-04-30T06:38:30.866ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
76742025-04-30T06:38:30.867ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e5f60ec-65a7-43c4-8bd1-1c85aaf15094", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76752025-04-30T06:38:30.867ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
76762025-04-30T06:38:30.869ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "615dba2f-aa0e-46e8-b16b-6804708d3015", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76772025-04-30T06:38:30.869ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
76782025-04-30T06:38:30.870ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1999b420-fb95-45cb-84ae-0252d50a2018", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76792025-04-30T06:38:30.870ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
76802025-04-30T06:38:30.872ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83dc0358-6baf-49a4-9243-9ce08fc46c3c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76812025-04-30T06:38:30.872ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
76822025-04-30T06:38:30.874ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aed77986-c921-499b-9755-91a42ceedea7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76832025-04-30T06:38:30.875ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
76842025-04-30T06:38:30.876ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b030f4f-ace9-4d38-bc75-270266f4bdb8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76852025-04-30T06:38:30.876ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
76862025-04-30T06:38:30.880ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9762f47b-edb9-48b1-b018-6fba44cf16f3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76872025-04-30T06:38:30.880ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
76882025-04-30T06:38:30.884ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2aeb5a07-7682-4732-8a98-cdc38b6e7352", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76892025-04-30T06:38:30.884ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
76902025-04-30T06:38:30.886ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ed2f92b-f087-4b5c-bed3-bd60a9a3cd43", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76912025-04-30T06:38:30.886ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
76922025-04-30T06:38:30.887ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe67064f-6329-431e-9eed-a0bca780d38c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76932025-04-30T06:38:30.887ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
76942025-04-30T06:38:30.888ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cd24df4-53f2-4fd1-b33a-82b0bbc8e8bd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76952025-04-30T06:38:30.888ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
76962025-04-30T06:38:30.889ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71831c1e-7fe5-4666-8c15-0e37d6e9bcc6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76972025-04-30T06:38:30.889ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
76982025-04-30T06:38:30.890ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6730874-0dcc-4815-b833-fe80534fa9cd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
76992025-04-30T06:38:30.890ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
77002025-04-30T06:38:30.892ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77012025-04-30T06:38:30.892ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77022025-04-30T06:38:30.892ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77032025-04-30T06:38:30.892ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
77042025-04-30T06:38:30.893ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77052025-04-30T06:38:30.893ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14d96957-0b6c-49f6-92d6-12406196ea03", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77062025-04-30T06:38:30.893ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
77072025-04-30T06:38:30.896ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c9c314b-29d9-4aa6-a73c-2dbaf4eef1a8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77082025-04-30T06:38:30.896ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
77092025-04-30T06:38:30.898ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77102025-04-30T06:38:30.908ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77112025-04-30T06:38:30.908ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77122025-04-30T06:38:30.908ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
77132025-04-30T06:38:30.909ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77142025-04-30T06:38:30.912ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "506db1ad-c668-4e61-98d7-9ca948897e79", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77152025-04-30T06:38:30.912ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
77162025-04-30T06:38:30.913ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77172025-04-30T06:38:30.913ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
77182025-04-30T06:38:30.913ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77192025-04-30T06:38:30.921ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb66a8b1-c72d-4464-8e17-21585e276163", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77202025-04-30T06:38:30.921ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
77212025-04-30T06:38:31.001ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2800833a-b8bd-4f62-87a5-5b09947edb7b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77222025-04-30T06:38:31.001ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
77232025-04-30T06:38:31.010ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
77242025-04-30T06:38:31.013ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bed16dd4-1c36-4c60-b50b-36fdb21ddad5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77252025-04-30T06:38:31.013ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
77262025-04-30T06:38:31.016ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06b65974-d7ca-47be-b2c0-57bc0e8c8106", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77272025-04-30T06:38:31.016ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
77282025-04-30T06:38:31.018ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "245e3d03-9764-479a-bc2e-bdd664bac4d7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77292025-04-30T06:38:31.018ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
77302025-04-30T06:38:31.020ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40e9bf07-8f61-44e9-b53c-e286c57f30d4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77312025-04-30T06:38:31.020ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
77322025-04-30T06:38:31.024ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9be389f4-a5f3-45e0-a233-735303a64d12", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77332025-04-30T06:38:31.024ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
77342025-04-30T06:38:31.025ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cc34e60-c3bf-4ad3-b616-789b96ca699d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77352025-04-30T06:38:31.025ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
77362025-04-30T06:38:31.026ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77372025-04-30T06:38:31.026ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77382025-04-30T06:38:31.026ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77392025-04-30T06:38:31.027ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77402025-04-30T06:38:31.033ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77412025-04-30T06:38:31.038ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
77422025-04-30T06:38:31.038ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
77432025-04-30T06:38:31.038ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77442025-04-30T06:38:31.038ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77452025-04-30T06:38:31.038ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77462025-04-30T06:38:31.039ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77472025-04-30T06:38:31.042ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77482025-04-30T06:38:31.047ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
77492025-04-30T06:38:31.047ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77502025-04-30T06:38:31.047ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77512025-04-30T06:38:31.048ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77522025-04-30T06:38:31.109ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
77532025-04-30T06:38:31.109ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
77542025-04-30T06:38:31.110ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77552025-04-30T06:38:31.110ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77562025-04-30T06:38:31.110ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77572025-04-30T06:38:31.110ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
77582025-04-30T06:38:31.111ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77592025-04-30T06:38:31.111ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:36769 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
77602025-04-30T06:38:31.111ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77612025-04-30T06:38:31.111ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77622025-04-30T06:38:31.112ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
77632025-04-30T06:38:31.113ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
77642025-04-30T06:38:31.114ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:36769 req_id = 49cb07d1-45e4-488b-ad3b-685d67616ab6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
77652025-04-30T06:38:31.114ZINFOtest_quotas (dropshot): request completed latency_us = 448 local_addr = [::1]:55655 method = GET remote_addr = [::1]:36769 req_id = 49cb07d1-45e4-488b-ad3b-685d67616ab6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
77662025-04-30T06:38:31.115ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "49cb07d1-45e4-488b-ad3b-685d67616ab6", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
77672025-04-30T06:38:31.118ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
77682025-04-30T06:38:31.119ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
77692025-04-30T06:38:31.119ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
77702025-04-30T06:38:31.121ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77712025-04-30T06:38:31.121ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77722025-04-30T06:38:31.121ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77732025-04-30T06:38:31.121ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77742025-04-30T06:38:31.122ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77752025-04-30T06:38:31.122ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77762025-04-30T06:38:31.132ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77772025-04-30T06:38:31.132ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77782025-04-30T06:38:31.132ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
77792025-04-30T06:38:31.133ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77802025-04-30T06:38:31.133ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
77812025-04-30T06:38:31.133ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
77822025-04-30T06:38:31.136ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77832025-04-30T06:38:31.141ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77842025-04-30T06:38:31.142ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77852025-04-30T06:38:31.142ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77862025-04-30T06:38:31.142ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77872025-04-30T06:38:31.146ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77882025-04-30T06:38:31.147ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
77892025-04-30T06:38:31.147ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77902025-04-30T06:38:31.147ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
77912025-04-30T06:38:31.147ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
77922025-04-30T06:38:31.189ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
77932025-04-30T06:38:31.189ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
77942025-04-30T06:38:31.189ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
77952025-04-30T06:38:31.189ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 486.384909ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
77962025-04-30T06:38:31.190ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
77972025-04-30T06:38:31.190ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77982025-04-30T06:38:31.190ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77992025-04-30T06:38:31.190ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78002025-04-30T06:38:31.190ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78012025-04-30T06:38:31.190ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78022025-04-30T06:38:31.193ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78032025-04-30T06:38:31.197ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
78042025-04-30T06:38:31.197ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78052025-04-30T06:38:31.197ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78062025-04-30T06:38:31.197ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78072025-04-30T06:38:31.200ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
78082025-04-30T06:38:31.200ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
78092025-04-30T06:38:31.201ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 1f ee 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63485 req_id = 62e9dcbb-8ea0-4627-8ed8-3ad361e84547
78102025-04-30T06:38:31.201ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8174,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63485 req_id = 62e9dcbb-8ea0-4627-8ed8-3ad361e84547
78112025-04-30T06:38:31.201ZDEBGtest_quotas (store): query key key = _mgs._tcp
78122025-04-30T06:38:31.201ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
78132025-04-30T06:38:31.201ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63485 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 62e9dcbb-8ea0-4627-8ed8-3ad361e84547
78142025-04-30T06:38:31.202ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3976, tv_nsec: 1490392 } })
78152025-04-30T06:38:31.202ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
78162025-04-30T06:38:31.203ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b9 17 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58272 req_id = facc4e24-a913-41a2-9188-da7dfb6c93b4
78172025-04-30T06:38:31.203ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47383,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58272 req_id = facc4e24-a913-41a2-9188-da7dfb6c93b4
78182025-04-30T06:38:31.204ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
78192025-04-30T06:38:31.204ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58272 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = facc4e24-a913-41a2-9188-da7dfb6c93b4
78202025-04-30T06:38:31.250ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
78212025-04-30T06:38:31.251ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78222025-04-30T06:38:31.251ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78232025-04-30T06:38:31.251ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78242025-04-30T06:38:31.251ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
78252025-04-30T06:38:31.252ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78262025-04-30T06:38:31.252ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
78272025-04-30T06:38:31.252ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6d f8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c m............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56963 req_id = adb8c979-9d0f-4233-9c33-1ce55a1b1b5b
78282025-04-30T06:38:31.252ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28152,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56963 req_id = adb8c979-9d0f-4233-9c33-1ce55a1b1b5b
78292025-04-30T06:38:31.253ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
78302025-04-30T06:38:31.253ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28152, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56963 req_id = adb8c979-9d0f-4233-9c33-1ce55a1b1b5b
78312025-04-30T06:38:31.253ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
78322025-04-30T06:38:31.254ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
78332025-04-30T06:38:31.254ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
78342025-04-30T06:38:31.255ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78352025-04-30T06:38:31.261ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78362025-04-30T06:38:31.261ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78372025-04-30T06:38:31.261ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
78382025-04-30T06:38:31.262ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78392025-04-30T06:38:31.263ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "50cdd11d-1003-49dc-be30-8680a63dce9a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78402025-04-30T06:38:31.263ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
78412025-04-30T06:38:31.264ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78422025-04-30T06:38:31.264ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
78432025-04-30T06:38:31.265ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78442025-04-30T06:38:31.266ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb0f5e30-6d6f-4642-a9c6-ddacf4134b6e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78452025-04-30T06:38:31.266ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
78462025-04-30T06:38:31.270ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7bc89c9-f69b-418c-8b56-51baa796fbac", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78472025-04-30T06:38:31.270ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
78482025-04-30T06:38:31.273ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c3d6151-9904-40c9-ae54-6acda7dbae9c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78492025-04-30T06:38:31.273ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
78502025-04-30T06:38:31.275ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb25991d-4785-461e-a57b-1a2cb87cb49d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78512025-04-30T06:38:31.276ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
78522025-04-30T06:38:31.278ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fc45590-9b2e-4466-be9d-e9957c4cee94", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78532025-04-30T06:38:31.278ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
78542025-04-30T06:38:31.283ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68007db3-0d05-4e3f-aa8c-64520277fa0c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78552025-04-30T06:38:31.283ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
78562025-04-30T06:38:31.288ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8875c5d7-0f2f-4964-86b2-277f3b7d161c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78572025-04-30T06:38:31.288ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
78582025-04-30T06:38:31.289ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51307c74-b2fd-4e23-a7bf-a07e0b33b337", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78592025-04-30T06:38:31.289ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
78602025-04-30T06:38:31.290ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d775483e-50de-4bd0-bbd7-ac670223ac4e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78612025-04-30T06:38:31.290ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
78622025-04-30T06:38:31.294ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
78632025-04-30T06:38:31.295ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0492b445-09c4-4f31-8c4c-fb9f41ae31ca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78642025-04-30T06:38:31.295ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
78652025-04-30T06:38:31.296ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9831b95b-663d-4020-8235-0f948b663eb0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78662025-04-30T06:38:31.297ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
78672025-04-30T06:38:31.298ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d73afafe-87da-4468-b44b-66a421d34e9a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78682025-04-30T06:38:31.298ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
78692025-04-30T06:38:31.300ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0885e34-58d3-4f18-ab0c-5e51302a8c57", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78702025-04-30T06:38:31.300ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
78712025-04-30T06:38:31.302ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f809b9d4-fa3c-4fd4-b90b-900461f79368", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78722025-04-30T06:38:31.302ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
78732025-04-30T06:38:31.304ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7fe4197-ba53-426c-b22d-ed2033535784", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78742025-04-30T06:38:31.304ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
78752025-04-30T06:38:31.306ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cef2a3a-0f80-4600-be58-c23fd3b50c40", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78762025-04-30T06:38:31.306ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
78772025-04-30T06:38:31.308ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c8a25b9-d145-4164-b8d5-b344bed59832", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78782025-04-30T06:38:31.308ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
78792025-04-30T06:38:31.310ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00bdcfac-3436-4974-b19c-a6541eae0520", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78802025-04-30T06:38:31.311ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
78812025-04-30T06:38:31.312ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e97e8b62-b79e-4f9b-bd92-ecd9d19b99c3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78822025-04-30T06:38:31.312ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
78832025-04-30T06:38:31.313ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "087c5536-6d89-4414-9175-7d53d3d19eeb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78842025-04-30T06:38:31.313ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
78852025-04-30T06:38:31.314ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6012b607-e573-440b-9201-94f512141b9c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78862025-04-30T06:38:31.314ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
78872025-04-30T06:38:31.315ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "039e3b57-cb3e-4094-82df-bffbe509cb9b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78882025-04-30T06:38:31.315ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
78892025-04-30T06:38:31.316ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08cd2095-cae9-42c0-99f2-f1b2d00c7535", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78902025-04-30T06:38:31.317ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
78912025-04-30T06:38:31.319ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "098c16e4-f429-4388-9644-37987e5630ee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78922025-04-30T06:38:31.319ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
78932025-04-30T06:38:31.321ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f1b9e74-7391-41f5-ad07-f0a34c57cbec", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78942025-04-30T06:38:31.321ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
78952025-04-30T06:38:31.323ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8272f5e9-866d-4175-9d51-bfa4d000a89b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78962025-04-30T06:38:31.323ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
78972025-04-30T06:38:31.325ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8decff4f-7cfb-407c-b8c5-c2bd968133c5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
78982025-04-30T06:38:31.325ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
78992025-04-30T06:38:31.327ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28a1ffb1-3ed9-4640-896d-d1a93a90cd03", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79002025-04-30T06:38:31.327ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
79012025-04-30T06:38:31.329ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0ffda3a-d71b-4ee0-9e57-8afb5d5d2c4d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79022025-04-30T06:38:31.329ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
79032025-04-30T06:38:31.330ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0aba2e2e-e5b2-4179-b9fd-dbcc93ee6edb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79042025-04-30T06:38:31.330ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
79052025-04-30T06:38:31.331ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8647f915-0982-4b3b-bb24-67a08283f20a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79062025-04-30T06:38:31.331ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
79072025-04-30T06:38:31.332ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7952dac-e612-477c-98c1-245237a1f38d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79082025-04-30T06:38:31.332ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
79092025-04-30T06:38:31.333ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74fb392e-92e9-43ce-8cd4-52c882e0142c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79102025-04-30T06:38:31.333ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
79112025-04-30T06:38:31.334ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "351d5c03-3eec-43e5-88ca-e0d0388d2749", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79122025-04-30T06:38:31.334ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
79132025-04-30T06:38:31.336ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b25f7f6-dc4d-4eb9-bcd4-420fca20250c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79142025-04-30T06:38:31.336ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
79152025-04-30T06:38:31.338ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6f4c750-2b85-45cb-919a-6ae97afbe295", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79162025-04-30T06:38:31.338ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
79172025-04-30T06:38:31.340ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d781f04-8e55-472e-941b-c5ded6415f84", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79182025-04-30T06:38:31.340ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
79192025-04-30T06:38:31.341ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a47913d4-85b0-460a-bc94-c71ab853c65d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79202025-04-30T06:38:31.341ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
79212025-04-30T06:38:31.343ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d59aed17-fde7-4bb1-a864-291145cc65f1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79222025-04-30T06:38:31.343ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
79232025-04-30T06:38:31.345ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79242025-04-30T06:38:31.345ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79252025-04-30T06:38:31.345ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79262025-04-30T06:38:31.345ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
79272025-04-30T06:38:31.346ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79282025-04-30T06:38:31.346ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "175adb87-5225-4455-94ad-ab228b77c2a3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79292025-04-30T06:38:31.347ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
79302025-04-30T06:38:31.348ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79312025-04-30T06:38:31.357ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79322025-04-30T06:38:31.357ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79332025-04-30T06:38:31.358ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
79342025-04-30T06:38:31.358ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79352025-04-30T06:38:31.359ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "360ad87b-8c0f-4e75-909c-f8b6a311e132", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79362025-04-30T06:38:31.359ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
79372025-04-30T06:38:31.360ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8477d741-4fa7-4008-926e-7c6164f91ef2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79382025-04-30T06:38:31.360ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
79392025-04-30T06:38:31.360ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79402025-04-30T06:38:31.360ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
79412025-04-30T06:38:31.361ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79422025-04-30T06:38:31.362ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26ff7e34-0bcd-4755-bcca-6d1d6d7ac689", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79432025-04-30T06:38:31.362ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
79442025-04-30T06:38:31.364ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc3c2cf7-ceaf-434d-a9b0-59fa496de2e0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79452025-04-30T06:38:31.364ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79462025-04-30T06:38:31.364ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79472025-04-30T06:38:31.364ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79482025-04-30T06:38:31.364ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79492025-04-30T06:38:31.365ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79502025-04-30T06:38:31.368ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79512025-04-30T06:38:31.374ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79522025-04-30T06:38:31.374ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79532025-04-30T06:38:31.374ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79542025-04-30T06:38:31.374ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79552025-04-30T06:38:31.374ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79562025-04-30T06:38:31.375ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79572025-04-30T06:38:31.378ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79582025-04-30T06:38:31.384ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
79592025-04-30T06:38:31.384ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79602025-04-30T06:38:31.384ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79612025-04-30T06:38:31.384ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79622025-04-30T06:38:31.434ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
79632025-04-30T06:38:31.434ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
79642025-04-30T06:38:31.434ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:37263 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
79652025-04-30T06:38:31.435ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:37263 req_id = d2ec3aef-3034-46a4-a1f8-0c9c78cfb4be sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79662025-04-30T06:38:31.436ZINFOtest_quotas (dropshot): request completed latency_us = 578 local_addr = [::1]:55655 method = GET remote_addr = [::1]:37263 req_id = d2ec3aef-3034-46a4-a1f8-0c9c78cfb4be response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
79672025-04-30T06:38:31.437ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2ec3aef-3034-46a4-a1f8-0c9c78cfb4be", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:30 GMT"} })
79682025-04-30T06:38:31.440ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79692025-04-30T06:38:31.440ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
79702025-04-30T06:38:31.440ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
79712025-04-30T06:38:31.442ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79722025-04-30T06:38:31.443ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79732025-04-30T06:38:31.443ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79742025-04-30T06:38:31.443ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79752025-04-30T06:38:31.443ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79762025-04-30T06:38:31.447ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79772025-04-30T06:38:31.452ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79782025-04-30T06:38:31.452ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79792025-04-30T06:38:31.452ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79802025-04-30T06:38:31.452ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79812025-04-30T06:38:31.464ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
79822025-04-30T06:38:31.483ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
79832025-04-30T06:38:31.483ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
79842025-04-30T06:38:31.484ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 294.095807ms iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
79852025-04-30T06:38:31.484ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
79862025-04-30T06:38:31.484ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
79872025-04-30T06:38:31.484ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79882025-04-30T06:38:31.484ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79892025-04-30T06:38:31.484ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79902025-04-30T06:38:31.485ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79912025-04-30T06:38:31.486ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79922025-04-30T06:38:31.493ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79932025-04-30T06:38:31.493ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79942025-04-30T06:38:31.493ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79952025-04-30T06:38:31.493ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79962025-04-30T06:38:31.497ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
79972025-04-30T06:38:31.497ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
79982025-04-30T06:38:31.498ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 98 86 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64310 req_id = d326e631-6e00-4186-8a02-c6492cec63a5
79992025-04-30T06:38:31.498ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39046,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64310 req_id = d326e631-6e00-4186-8a02-c6492cec63a5
80002025-04-30T06:38:31.498ZDEBGtest_quotas (store): query key key = _mgs._tcp
80012025-04-30T06:38:31.498ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80022025-04-30T06:38:31.498ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64310 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d326e631-6e00-4186-8a02-c6492cec63a5
80032025-04-30T06:38:31.499ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3976, tv_nsec: 298035819 } })
80042025-04-30T06:38:31.499ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
80052025-04-30T06:38:31.499ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 31 bd 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 1...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62089 req_id = 54cf0ea1-f214-4237-a231-4f6fc42d46a5
80062025-04-30T06:38:31.499ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12733,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62089 req_id = 54cf0ea1-f214-4237-a231-4f6fc42d46a5
80072025-04-30T06:38:31.499ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80082025-04-30T06:38:31.499ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62089 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 54cf0ea1-f214-4237-a231-4f6fc42d46a5
80092025-04-30T06:38:31.541ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
80102025-04-30T06:38:31.541ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80112025-04-30T06:38:31.541ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80122025-04-30T06:38:31.542ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80132025-04-30T06:38:31.542ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
80142025-04-30T06:38:31.543ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80152025-04-30T06:38:31.544ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 39 2c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 9,..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55588 req_id = af5b2ed3-6aec-4134-8eb6-f14f6397da24
80162025-04-30T06:38:31.544ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14636,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55588 req_id = af5b2ed3-6aec-4134-8eb6-f14f6397da24
80172025-04-30T06:38:31.544ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
80182025-04-30T06:38:31.545ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14636, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55588 req_id = af5b2ed3-6aec-4134-8eb6-f14f6397da24
80192025-04-30T06:38:31.546ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
80202025-04-30T06:38:31.546ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
80212025-04-30T06:38:31.546ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
80222025-04-30T06:38:31.546ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80232025-04-30T06:38:31.553ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80242025-04-30T06:38:31.553ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80252025-04-30T06:38:31.553ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
80262025-04-30T06:38:31.554ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80272025-04-30T06:38:31.555ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e784c8ce-6c9e-4140-bc20-b954905e02a9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80282025-04-30T06:38:31.555ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
80292025-04-30T06:38:31.556ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b3f4f02-e913-4a3d-93e8-782b28b4f426", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80302025-04-30T06:38:31.556ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
80312025-04-30T06:38:31.557ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80322025-04-30T06:38:31.557ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
80332025-04-30T06:38:31.557ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80342025-04-30T06:38:31.560ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61592a9d-4fff-4b0c-afe3-206f11286c26", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80352025-04-30T06:38:31.560ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
80362025-04-30T06:38:31.572ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "711ec7c5-7ab5-4812-a3bf-21af86b32e7a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80372025-04-30T06:38:31.572ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
80382025-04-30T06:38:31.578ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fa4ad34-ebbe-47f1-a396-091dce015d53", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80392025-04-30T06:38:31.578ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
80402025-04-30T06:38:31.582ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18057506-7f75-4108-988a-af4f3d92cfde", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80412025-04-30T06:38:31.582ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
80422025-04-30T06:38:31.593ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4d56867-ddfe-459b-b8cc-d04dc9c15c31", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80432025-04-30T06:38:31.593ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
80442025-04-30T06:38:31.597ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "677cd761-fe84-497f-ae58-ebdad65a560e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80452025-04-30T06:38:31.597ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
80462025-04-30T06:38:31.599ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8ab15cb-593c-471c-bba1-0309aa6d27b1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80472025-04-30T06:38:31.599ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
80482025-04-30T06:38:31.602ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb1dca6e-145b-43d4-8635-06bd34d8067f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80492025-04-30T06:38:31.602ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
80502025-04-30T06:38:31.607ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46e55213-1101-400d-8028-58543ccabf6a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80512025-04-30T06:38:31.608ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
80522025-04-30T06:38:31.609ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
80532025-04-30T06:38:31.611ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cef373c4-a2ca-4dd9-bcdd-f3a83043ecd9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80542025-04-30T06:38:31.611ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
80552025-04-30T06:38:31.613ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b318626e-3343-400f-848f-4a95dd3a1bb2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80562025-04-30T06:38:31.613ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
80572025-04-30T06:38:31.615ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c85b1ee1-a27e-4ada-bab0-918493d2ac4c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80582025-04-30T06:38:31.615ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
80592025-04-30T06:38:31.617ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37ae7572-24ae-4b82-afd7-d1d1a7e643b8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80602025-04-30T06:38:31.617ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
80612025-04-30T06:38:31.619ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d5078e7-986d-494c-aa5e-d8d55a1148b0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80622025-04-30T06:38:31.619ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
80632025-04-30T06:38:31.622ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe137ac9-06a2-44ad-b109-7bb77b1e2bf7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80642025-04-30T06:38:31.622ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
80652025-04-30T06:38:31.625ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30f8cfb8-1c6c-409b-828b-2b0aa3a177a0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80662025-04-30T06:38:31.625ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
80672025-04-30T06:38:31.628ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bd1b0f6-099f-47ee-b838-609485279d2d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80682025-04-30T06:38:31.628ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
80692025-04-30T06:38:31.629ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87f5e332-a96a-4163-afaf-7fdbe13e6250", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80702025-04-30T06:38:31.630ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
80712025-04-30T06:38:31.631ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bebe427-3900-4491-a55c-211f017842ae", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80722025-04-30T06:38:31.631ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
80732025-04-30T06:38:31.633ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2712399-3665-4ad8-82dc-478324e72e49", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80742025-04-30T06:38:31.634ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
80752025-04-30T06:38:31.636ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae3e7675-3ba7-460c-b031-b2fc2edaa261", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80762025-04-30T06:38:31.637ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
80772025-04-30T06:38:31.638ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95d0ff01-8caa-4e87-990b-cc47c30086c9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80782025-04-30T06:38:31.639ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
80792025-04-30T06:38:31.641ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b4db030-3313-41e1-b395-6deebf391155", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80802025-04-30T06:38:31.641ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
80812025-04-30T06:38:31.643ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01e5c765-28a4-416e-9191-3b25c5351bfe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80822025-04-30T06:38:31.643ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
80832025-04-30T06:38:31.645ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e17a40e9-15a8-476b-8c12-822849a441e1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80842025-04-30T06:38:31.645ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
80852025-04-30T06:38:31.647ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1ec51f0-cf0a-4a7d-800f-e2f68477e48a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80862025-04-30T06:38:31.647ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
80872025-04-30T06:38:31.649ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9ab321a-e424-4501-8161-76bf206d7026", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80882025-04-30T06:38:31.651ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
80892025-04-30T06:38:31.652ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e38368f-0561-4b3c-9eb6-97d04622d3e1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80902025-04-30T06:38:31.652ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
80912025-04-30T06:38:31.657ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "261ae441-ba23-4407-8b74-6077c8b1503d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80922025-04-30T06:38:31.657ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
80932025-04-30T06:38:31.659ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80942025-04-30T06:38:31.659ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80952025-04-30T06:38:31.659ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80962025-04-30T06:38:31.659ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
80972025-04-30T06:38:31.660ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80982025-04-30T06:38:31.661ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76d33779-8861-4a3c-950e-7fe4bf57d7eb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
80992025-04-30T06:38:31.661ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
81002025-04-30T06:38:31.667ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81012025-04-30T06:38:31.692ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81022025-04-30T06:38:31.692ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81032025-04-30T06:38:31.692ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
81042025-04-30T06:38:31.693ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81052025-04-30T06:38:31.697ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a745837-c0a7-4796-aa4d-bd04100c576d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81062025-04-30T06:38:31.697ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
81072025-04-30T06:38:31.697ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81082025-04-30T06:38:31.698ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
81092025-04-30T06:38:31.699ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81102025-04-30T06:38:31.710ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "845a23a4-1f79-4ddf-93ef-f615a4235df8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81112025-04-30T06:38:31.710ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
81122025-04-30T06:38:31.718ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "697f7bce-5bbc-448f-a3b8-3b55257076bb", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81132025-04-30T06:38:31.718ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
81142025-04-30T06:38:31.723ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "851bf6c5-fd94-4097-8ef3-6c41a869e991", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81152025-04-30T06:38:31.723ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
81162025-04-30T06:38:31.785ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
81172025-04-30T06:38:31.786ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81182025-04-30T06:38:31.796ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d46735e5-38ce-43f2-87fc-54037d111acc uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81192025-04-30T06:38:31.797ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d46735e5-38ce-43f2-87fc-54037d111acc resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81202025-04-30T06:38:31.797ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d46735e5-38ce-43f2-87fc-54037d111acc roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81212025-04-30T06:38:31.801ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d46735e5-38ce-43f2-87fc-54037d111acc resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81222025-04-30T06:38:31.805ZINFOtest_quotas (dropshot_internal): request completed latency_us = 9464 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d46735e5-38ce-43f2-87fc-54037d111acc response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81232025-04-30T06:38:31.806ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "d46735e5-38ce-43f2-87fc-54037d111acc", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81242025-04-30T06:38:31.807ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
81252025-04-30T06:38:31.807ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03c8308e-16d9-4ed7-a81d-db5987af4c91", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81262025-04-30T06:38:31.807ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
81272025-04-30T06:38:31.808ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = c746ffd1-9bf7-463b-af86-3105b6b7aeb4 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
81282025-04-30T06:38:31.809ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = c746ffd1-9bf7-463b-af86-3105b6b7aeb4 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
81292025-04-30T06:38:31.809ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = c746ffd1-9bf7-463b-af86-3105b6b7aeb4 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
81302025-04-30T06:38:31.810ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = c746ffd1-9bf7-463b-af86-3105b6b7aeb4 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
81312025-04-30T06:38:31.814ZINFOtest_quotas (dropshot_internal): request completed latency_us = 5948 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = c746ffd1-9bf7-463b-af86-3105b6b7aeb4 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
81322025-04-30T06:38:31.816ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "c746ffd1-9bf7-463b-af86-3105b6b7aeb4", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81332025-04-30T06:38:31.816ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
81342025-04-30T06:38:31.816ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81352025-04-30T06:38:31.816ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
81362025-04-30T06:38:31.835ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "207c1e3c-7981-4ba1-89bb-153288b92f0e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81372025-04-30T06:38:31.836ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
81382025-04-30T06:38:31.837ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
81392025-04-30T06:38:31.841ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dbe9c53-3c9f-40b7-bda3-0c1104d46a30", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81402025-04-30T06:38:31.841ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
81412025-04-30T06:38:31.842ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a448a66-346d-46f5-aab2-a9b4c42d7273", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81422025-04-30T06:38:31.843ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
81432025-04-30T06:38:31.845ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eed76395-bfd9-4c88-b31e-1ce5d68ebed3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81442025-04-30T06:38:31.845ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
81452025-04-30T06:38:31.846ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "960568b1-ee53-45d2-9e54-7651a55b6384", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81462025-04-30T06:38:31.846ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
81472025-04-30T06:38:31.847ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8154e259-dec2-4c76-a10e-f03141d6d8b1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81482025-04-30T06:38:31.847ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
81492025-04-30T06:38:31.848ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "868ed9dc-98bf-421a-96a1-76b6843ecbea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81502025-04-30T06:38:31.848ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
81512025-04-30T06:38:31.849ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf02022e-46a7-4098-9666-d689a299cb39", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81522025-04-30T06:38:31.849ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81532025-04-30T06:38:31.850ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81542025-04-30T06:38:31.850ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81552025-04-30T06:38:31.850ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81562025-04-30T06:38:31.850ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81572025-04-30T06:38:31.851ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81582025-04-30T06:38:31.851ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81592025-04-30T06:38:31.852ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81602025-04-30T06:38:31.853ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81612025-04-30T06:38:31.853ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81622025-04-30T06:38:31.853ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81632025-04-30T06:38:31.861ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81642025-04-30T06:38:31.866ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81652025-04-30T06:38:31.866ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81662025-04-30T06:38:31.866ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81672025-04-30T06:38:31.866ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81682025-04-30T06:38:31.866ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81692025-04-30T06:38:31.866ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81702025-04-30T06:38:31.873ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81712025-04-30T06:38:31.878ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
81722025-04-30T06:38:31.878ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81732025-04-30T06:38:31.878ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81742025-04-30T06:38:31.878ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81752025-04-30T06:38:31.891ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81762025-04-30T06:38:31.891ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81772025-04-30T06:38:31.891ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81782025-04-30T06:38:31.891ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
81792025-04-30T06:38:31.893ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81802025-04-30T06:38:31.978ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
81812025-04-30T06:38:31.978ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
81822025-04-30T06:38:31.978ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:40785 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
81832025-04-30T06:38:31.980ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:40785 req_id = c1ca21cc-5c8d-46e1-9959-e08b182478fb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81842025-04-30T06:38:31.980ZINFOtest_quotas (dropshot): request completed latency_us = 541 local_addr = [::1]:55655 method = GET remote_addr = [::1]:40785 req_id = c1ca21cc-5c8d-46e1-9959-e08b182478fb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81852025-04-30T06:38:31.980ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81862025-04-30T06:38:31.987ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81872025-04-30T06:38:31.987ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81882025-04-30T06:38:31.987ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
81892025-04-30T06:38:31.987ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81902025-04-30T06:38:31.988ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1ca21cc-5c8d-46e1-9959-e08b182478fb", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
81912025-04-30T06:38:31.990ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81922025-04-30T06:38:31.990ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81932025-04-30T06:38:31.990ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
81942025-04-30T06:38:31.992ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81952025-04-30T06:38:31.992ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81962025-04-30T06:38:31.992ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81972025-04-30T06:38:31.992ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81982025-04-30T06:38:31.993ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81992025-04-30T06:38:31.998ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82002025-04-30T06:38:31.998ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
82012025-04-30T06:38:31.999ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82022025-04-30T06:38:32.001ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82032025-04-30T06:38:32.005ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
82042025-04-30T06:38:32.005ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82052025-04-30T06:38:32.005ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82062025-04-30T06:38:32.006ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82072025-04-30T06:38:32.103ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82082025-04-30T06:38:32.104ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82092025-04-30T06:38:32.104ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
82102025-04-30T06:38:32.104ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
82112025-04-30T06:38:32.109ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
82122025-04-30T06:38:32.109ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
82132025-04-30T06:38:32.109ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 625.571631ms iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
82142025-04-30T06:38:32.109ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
82152025-04-30T06:38:32.109ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
82162025-04-30T06:38:32.110ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82172025-04-30T06:38:32.110ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82182025-04-30T06:38:32.110ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82192025-04-30T06:38:32.110ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82202025-04-30T06:38:32.112ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
82212025-04-30T06:38:32.112ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
82222025-04-30T06:38:32.112ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
82232025-04-30T06:38:32.114ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82242025-04-30T06:38:32.118ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
82252025-04-30T06:38:32.118ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82262025-04-30T06:38:32.118ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82272025-04-30T06:38:32.119ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82282025-04-30T06:38:32.119ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
82292025-04-30T06:38:32.119ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82302025-04-30T06:38:32.122ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
82312025-04-30T06:38:32.122ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
82322025-04-30T06:38:32.123ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9f 86 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53885 req_id = d4027b47-0710-480a-82d1-87c3e5eebe5c
82332025-04-30T06:38:32.123ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40838,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53885 req_id = d4027b47-0710-480a-82d1-87c3e5eebe5c
82342025-04-30T06:38:32.124ZDEBGtest_quotas (store): query key key = _mgs._tcp
82352025-04-30T06:38:32.124ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82362025-04-30T06:38:32.124ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53885 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d4027b47-0710-480a-82d1-87c3e5eebe5c
82372025-04-30T06:38:32.124ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3976, tv_nsec: 923538012 } })
82382025-04-30T06:38:32.124ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
82392025-04-30T06:38:32.125ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1c 99 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52663 req_id = 5285ba62-2ae5-4fe6-afbd-3ede76bf9cde
82402025-04-30T06:38:32.125ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7321,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52663 req_id = 5285ba62-2ae5-4fe6-afbd-3ede76bf9cde
82412025-04-30T06:38:32.125ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82422025-04-30T06:38:32.125ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52663 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5285ba62-2ae5-4fe6-afbd-3ede76bf9cde
82432025-04-30T06:38:32.165ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
82442025-04-30T06:38:32.165ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82452025-04-30T06:38:32.165ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82462025-04-30T06:38:32.166ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82472025-04-30T06:38:32.174ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
82482025-04-30T06:38:32.175ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82492025-04-30T06:38:32.176ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ff b2 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60953 req_id = 78e7ef75-1e6d-49cf-86b2-a4d1fa9f7d67
82502025-04-30T06:38:32.176ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65458,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60953 req_id = 78e7ef75-1e6d-49cf-86b2-a4d1fa9f7d67
82512025-04-30T06:38:32.176ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
82522025-04-30T06:38:32.176ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65458, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60953 req_id = 78e7ef75-1e6d-49cf-86b2-a4d1fa9f7d67
82532025-04-30T06:38:32.178ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
82542025-04-30T06:38:32.178ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
82552025-04-30T06:38:32.178ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
82562025-04-30T06:38:32.178ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82572025-04-30T06:38:32.183ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82582025-04-30T06:38:32.183ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82592025-04-30T06:38:32.183ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
82602025-04-30T06:38:32.184ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82612025-04-30T06:38:32.186ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "80bd7e87-c0da-4a62-8dec-f999c2a50b1e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82622025-04-30T06:38:32.186ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
82632025-04-30T06:38:32.193ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82642025-04-30T06:38:32.197ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
82652025-04-30T06:38:32.197ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82662025-04-30T06:38:32.197ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc19a1a4-2244-4823-aa92-4a56da478891", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82672025-04-30T06:38:32.197ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
82682025-04-30T06:38:32.197ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20e31551-9baf-4620-b747-4e31f4dfed49", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82692025-04-30T06:38:32.197ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
82702025-04-30T06:38:32.209ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
82712025-04-30T06:38:32.209ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3cb92c8-bf5c-4e33-9fa0-cfbb322bb6ed", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82722025-04-30T06:38:32.209ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
82732025-04-30T06:38:32.211ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed234b6f-837d-4cf5-8d4d-1b608bf7ee30", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82742025-04-30T06:38:32.212ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
82752025-04-30T06:38:32.213ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04ce3cce-70a6-4f53-bea2-0b2f8e682d32", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82762025-04-30T06:38:32.213ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
82772025-04-30T06:38:32.216ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6936265d-c31c-48b4-9db9-90a6badb797b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82782025-04-30T06:38:32.216ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
82792025-04-30T06:38:32.226ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b337e8f-98b6-47e9-940a-175fe7372c8a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82802025-04-30T06:38:32.226ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
82812025-04-30T06:38:32.228ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55e3fc5d-e76a-420c-af89-b9a80854bfeb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82822025-04-30T06:38:32.228ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
82832025-04-30T06:38:32.230ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed21a2f7-69a6-450c-9591-7589c492cbe3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82842025-04-30T06:38:32.230ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
82852025-04-30T06:38:32.232ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e3947cf-20e0-4cc0-aa9b-44beb3565eb4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82862025-04-30T06:38:32.232ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
82872025-04-30T06:38:32.233ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6b28fac-59bb-490a-bfb6-cf5b0afcfc72", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82882025-04-30T06:38:32.233ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
82892025-04-30T06:38:32.235ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bfcc42e-17ef-4f54-920d-b0c79c57f6ab", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82902025-04-30T06:38:32.235ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
82912025-04-30T06:38:32.238ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
82922025-04-30T06:38:32.239ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6424232a-9716-4cfb-aa9a-64b72dc11fc5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82932025-04-30T06:38:32.240ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
82942025-04-30T06:38:32.241ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "615cb00d-f9b4-4fca-8d11-985d23d3d753", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82952025-04-30T06:38:32.241ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
82962025-04-30T06:38:32.243ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "78242aa9-539f-43d1-9996-80d41f7cddae", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82972025-04-30T06:38:32.243ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
82982025-04-30T06:38:32.246ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ae779ab-7e7f-404f-834e-49053c4b0a8c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
82992025-04-30T06:38:32.246ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
83002025-04-30T06:38:32.247ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f1f657a-e9a2-4537-976e-9eb940d59feb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83012025-04-30T06:38:32.248ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
83022025-04-30T06:38:32.249ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0e03c1f-c9ec-462e-b194-0ebc495b61f6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83032025-04-30T06:38:32.250ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
83042025-04-30T06:38:32.250ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dc3895f-524d-4e24-bbac-d83dd0806331", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83052025-04-30T06:38:32.251ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
83062025-04-30T06:38:32.251ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02724b16-0b3b-4784-aaba-d38e4cd1a919", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83072025-04-30T06:38:32.252ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
83082025-04-30T06:38:32.252ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fb47be4-ef7c-464d-8ef0-eb47e3ff4e40", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83092025-04-30T06:38:32.253ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
83102025-04-30T06:38:32.254ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4cbe7d7-747d-4c6f-8ae4-dbf5ba28f67e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83112025-04-30T06:38:32.255ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
83122025-04-30T06:38:32.257ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eac651d4-f5f3-4ae9-be14-ec082fdd02f9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83132025-04-30T06:38:32.257ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
83142025-04-30T06:38:32.259ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "928fc980-a56c-4c36-9831-51ff11cea175", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83152025-04-30T06:38:32.259ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
83162025-04-30T06:38:32.263ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "449f06f6-38cc-4d95-beda-6ea587dd6b62", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83172025-04-30T06:38:32.263ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
83182025-04-30T06:38:32.265ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b3d1af2-d4ff-4ce9-8eec-a65c83a13139", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83192025-04-30T06:38:32.265ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
83202025-04-30T06:38:32.268ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fa875e4-fc14-4b33-ba08-593471b2698d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83212025-04-30T06:38:32.268ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
83222025-04-30T06:38:32.270ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c9905a5-cccd-4b7f-9897-cecc6086d38f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83232025-04-30T06:38:32.270ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
83242025-04-30T06:38:32.272ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "677f9eac-f291-42e7-bc48-d5d3a929401b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83252025-04-30T06:38:32.272ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
83262025-04-30T06:38:32.273ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39d11b4c-0d85-48d8-87e1-ed9243d0831f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83272025-04-30T06:38:32.273ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
83282025-04-30T06:38:32.274ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24565590-5090-4171-8662-57b13267d75d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83292025-04-30T06:38:32.274ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
83302025-04-30T06:38:32.275ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1587726-4947-484b-aa19-066bb798450b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83312025-04-30T06:38:32.275ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
83322025-04-30T06:38:32.277ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "116012b6-fa73-46b3-8ce2-3a4d0fcdb0b9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83332025-04-30T06:38:32.277ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
83342025-04-30T06:38:32.278ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ffb7ec2-95a7-4481-97c8-7b24ce17ffb1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83352025-04-30T06:38:32.278ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
83362025-04-30T06:38:32.280ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52166f54-5a86-4204-9da4-77186bf734d6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83372025-04-30T06:38:32.280ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
83382025-04-30T06:38:32.282ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a43ad4f-b984-4fa7-ac12-869551390664", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83392025-04-30T06:38:32.282ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
83402025-04-30T06:38:32.284ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e24e538-bf45-497f-8e10-88806063a400", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83412025-04-30T06:38:32.284ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
83422025-04-30T06:38:32.286ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "875be6a2-70f8-4031-a757-4b38dabb5e0d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83432025-04-30T06:38:32.286ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
83442025-04-30T06:38:32.288ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36ee75e1-cf8d-4e7f-9d85-49a3afadaa7a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83452025-04-30T06:38:32.288ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
83462025-04-30T06:38:32.289ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83472025-04-30T06:38:32.289ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83482025-04-30T06:38:32.289ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83492025-04-30T06:38:32.289ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
83502025-04-30T06:38:32.290ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83512025-04-30T06:38:32.291ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9095295e-44f1-47f6-8241-8bd0bf9711c6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83522025-04-30T06:38:32.291ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
83532025-04-30T06:38:32.292ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22e79955-24e8-4da7-99f2-df82aaafa55b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83542025-04-30T06:38:32.292ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
83552025-04-30T06:38:32.293ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83562025-04-30T06:38:32.302ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83572025-04-30T06:38:32.302ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83582025-04-30T06:38:32.302ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
83592025-04-30T06:38:32.303ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83602025-04-30T06:38:32.304ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08f7ac3f-c698-4232-b5fe-7a84b7b0cc0c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83612025-04-30T06:38:32.304ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
83622025-04-30T06:38:32.306ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b6daafb-2418-4e4f-b7d8-210311818ce4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83632025-04-30T06:38:32.306ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
83642025-04-30T06:38:32.307ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83652025-04-30T06:38:32.307ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
83662025-04-30T06:38:32.307ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83672025-04-30T06:38:32.309ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1168bc63-9374-48d2-8e32-4e3eef746847", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83682025-04-30T06:38:32.309ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83692025-04-30T06:38:32.309ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83702025-04-30T06:38:32.310ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83712025-04-30T06:38:32.310ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83722025-04-30T06:38:32.310ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83732025-04-30T06:38:32.314ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83742025-04-30T06:38:32.323ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83752025-04-30T06:38:32.323ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83762025-04-30T06:38:32.323ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83772025-04-30T06:38:32.323ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83782025-04-30T06:38:32.323ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83792025-04-30T06:38:32.325ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83802025-04-30T06:38:32.328ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83812025-04-30T06:38:32.334ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83822025-04-30T06:38:32.334ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83832025-04-30T06:38:32.334ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83842025-04-30T06:38:32.335ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83852025-04-30T06:38:32.421ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
83862025-04-30T06:38:32.421ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
83872025-04-30T06:38:32.421ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:53489 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
83882025-04-30T06:38:32.423ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:53489 req_id = 549f4e02-510e-4a5b-8845-1c157d821830 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83892025-04-30T06:38:32.423ZINFOtest_quotas (dropshot): request completed latency_us = 625 local_addr = [::1]:55655 method = GET remote_addr = [::1]:53489 req_id = 549f4e02-510e-4a5b-8845-1c157d821830 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83902025-04-30T06:38:32.423ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "549f4e02-510e-4a5b-8845-1c157d821830", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:31 GMT"} })
83912025-04-30T06:38:32.425ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83922025-04-30T06:38:32.425ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83932025-04-30T06:38:32.425ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
83942025-04-30T06:38:32.427ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83952025-04-30T06:38:32.427ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83962025-04-30T06:38:32.427ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83972025-04-30T06:38:32.427ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83982025-04-30T06:38:32.428ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83992025-04-30T06:38:32.457ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84002025-04-30T06:38:32.471ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84012025-04-30T06:38:32.471ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84022025-04-30T06:38:32.471ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84032025-04-30T06:38:32.472ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84042025-04-30T06:38:32.484ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
84052025-04-30T06:38:32.539ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84062025-04-30T06:38:32.540ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84072025-04-30T06:38:32.540ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84082025-04-30T06:38:32.540ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
84092025-04-30T06:38:32.541ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84102025-04-30T06:38:32.549ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84112025-04-30T06:38:32.555ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84122025-04-30T06:38:32.555ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84132025-04-30T06:38:32.555ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
84142025-04-30T06:38:32.556ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84152025-04-30T06:38:32.561ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
84162025-04-30T06:38:32.562ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
84172025-04-30T06:38:32.562ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 452.560858ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
84182025-04-30T06:38:32.562ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
84192025-04-30T06:38:32.562ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84202025-04-30T06:38:32.562ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84212025-04-30T06:38:32.562ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84222025-04-30T06:38:32.562ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84232025-04-30T06:38:32.563ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84242025-04-30T06:38:32.564ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84252025-04-30T06:38:32.564ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
84262025-04-30T06:38:32.564ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84272025-04-30T06:38:32.567ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84282025-04-30T06:38:32.571ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84292025-04-30T06:38:32.571ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84302025-04-30T06:38:32.571ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84312025-04-30T06:38:32.572ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84322025-04-30T06:38:32.575ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
84332025-04-30T06:38:32.576ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
84342025-04-30T06:38:32.577ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 97 1e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55922 req_id = 485e81cb-a3ae-4ce9-b7fe-26d867e1a59f
84352025-04-30T06:38:32.577ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38686,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55922 req_id = 485e81cb-a3ae-4ce9-b7fe-26d867e1a59f
84362025-04-30T06:38:32.578ZDEBGtest_quotas (store): query key key = _mgs._tcp
84372025-04-30T06:38:32.578ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84382025-04-30T06:38:32.578ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55922 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 485e81cb-a3ae-4ce9-b7fe-26d867e1a59f
84392025-04-30T06:38:32.580ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3977, tv_nsec: 378366618 } })
84402025-04-30T06:38:32.580ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
84412025-04-30T06:38:32.581ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a7 3e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .>..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59811 req_id = 00eeb1f4-9cd9-4b36-aecc-4b9ac95053d7
84422025-04-30T06:38:32.581ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42814,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59811 req_id = 00eeb1f4-9cd9-4b36-aecc-4b9ac95053d7
84432025-04-30T06:38:32.581ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84442025-04-30T06:38:32.581ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59811 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 00eeb1f4-9cd9-4b36-aecc-4b9ac95053d7
84452025-04-30T06:38:32.713ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
84462025-04-30T06:38:32.714ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 22 b5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c "............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54202 req_id = 1bbafa5a-0037-4964-be93-e2adbd4adb86
84472025-04-30T06:38:32.714ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8885,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54202 req_id = 1bbafa5a-0037-4964-be93-e2adbd4adb86
84482025-04-30T06:38:32.715ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
84492025-04-30T06:38:32.715ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8885, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54202 req_id = 1bbafa5a-0037-4964-be93-e2adbd4adb86
84502025-04-30T06:38:32.715ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
84512025-04-30T06:38:32.715ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
84522025-04-30T06:38:32.715ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
84532025-04-30T06:38:32.717ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0ae33e0-0c57-4270-bf0b-078d40cd10c7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84542025-04-30T06:38:32.718ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
84552025-04-30T06:38:32.719ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4725725f-b35e-4ff0-8265-894f42b50a5f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84562025-04-30T06:38:32.719ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
84572025-04-30T06:38:32.721ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82a86fd9-bf75-439b-8fb3-f014242ff79c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84582025-04-30T06:38:32.721ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
84592025-04-30T06:38:32.723ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b630cc1-0714-4fd9-adb5-ac6724ca6eb0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84602025-04-30T06:38:32.723ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
84612025-04-30T06:38:32.725ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76d38b45-66d0-4946-8606-4a7f526e12be", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84622025-04-30T06:38:32.725ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
84632025-04-30T06:38:32.729ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73a5b50a-5da4-419a-b5f6-a39af2a2b4f6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84642025-04-30T06:38:32.729ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
84652025-04-30T06:38:32.735ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41ead139-c69d-446d-8c22-dcf08b9f0eb0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84662025-04-30T06:38:32.735ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
84672025-04-30T06:38:32.747ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71038830-da44-4721-b7f8-2b2dc2a91c38", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84682025-04-30T06:38:32.747ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
84692025-04-30T06:38:32.750ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ed8f3f2-ba30-428d-a8d2-a5bd47c0346c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84702025-04-30T06:38:32.750ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
84712025-04-30T06:38:32.754ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62251a5c-ebae-4518-b36f-55860eb09f12", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84722025-04-30T06:38:32.754ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
84732025-04-30T06:38:32.757ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
84742025-04-30T06:38:32.759ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89628007-af3b-41ad-8d53-4188892eb5f9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84752025-04-30T06:38:32.759ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
84762025-04-30T06:38:32.761ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca041853-1341-4d9c-af19-dac72613ef3a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84772025-04-30T06:38:32.761ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
84782025-04-30T06:38:32.763ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f53ef673-be98-453f-b954-1d6c8dae7b17", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84792025-04-30T06:38:32.763ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
84802025-04-30T06:38:32.765ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc258a68-8d6a-4f77-9c0e-28c8b738e479", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84812025-04-30T06:38:32.765ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
84822025-04-30T06:38:32.767ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b8e0d60-8e7a-495c-a9df-882bc1589fce", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84832025-04-30T06:38:32.767ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
84842025-04-30T06:38:32.769ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44d5a0a9-4379-41e8-8a39-ab8301b0ad7b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84852025-04-30T06:38:32.769ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
84862025-04-30T06:38:32.771ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e4b8602-6a2b-4949-a221-f0f4f4a0cde8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84872025-04-30T06:38:32.771ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
84882025-04-30T06:38:32.786ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aecd71f4-e5c3-4f7a-8d7e-720a0ae69135", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84892025-04-30T06:38:32.787ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
84902025-04-30T06:38:32.788ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "318d117d-d94c-4184-9632-e58624d42854", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84912025-04-30T06:38:32.788ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
84922025-04-30T06:38:32.792ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d225e3c0-59a2-4ecf-9a3f-dffad8f60ad5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84932025-04-30T06:38:32.792ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
84942025-04-30T06:38:32.792ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfafa527-428f-4a6c-a04a-faf059479123", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84952025-04-30T06:38:32.792ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
84962025-04-30T06:38:32.792ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d72a5eb5-c13e-4a21-846e-2c15ccbb56e0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84972025-04-30T06:38:32.792ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
84982025-04-30T06:38:32.792ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e3217c5-b819-43fb-a5f9-ebe3f7a9094d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
84992025-04-30T06:38:32.792ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
85002025-04-30T06:38:32.793ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0c0ac6c-3c81-4aa2-a4cb-e1ca79965d89", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85012025-04-30T06:38:32.793ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
85022025-04-30T06:38:32.795ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ead81a37-ca60-40ca-b278-5201cc8d0a2a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85032025-04-30T06:38:32.795ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
85042025-04-30T06:38:32.797ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbd622a3-ee15-441e-bda1-205c046a92be", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85052025-04-30T06:38:32.798ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
85062025-04-30T06:38:32.800ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8ddb19c-02da-466d-a5e0-a13560c81975", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85072025-04-30T06:38:32.800ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
85082025-04-30T06:38:32.802ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a69ac03b-b3e4-43ac-b6ee-a4851ee50139", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85092025-04-30T06:38:32.802ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
85102025-04-30T06:38:32.804ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcc92694-0280-49ec-bef3-83af7c629ae8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85112025-04-30T06:38:32.804ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
85122025-04-30T06:38:32.808ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db632c2e-8865-44dd-bfe1-92812e9b0e61", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85132025-04-30T06:38:32.808ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
85142025-04-30T06:38:32.809ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13deb3cc-edee-4487-a1dd-eb4182cd27c6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85152025-04-30T06:38:32.809ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
85162025-04-30T06:38:32.809ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85172025-04-30T06:38:32.809ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85182025-04-30T06:38:32.809ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85192025-04-30T06:38:32.809ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
85202025-04-30T06:38:32.810ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85212025-04-30T06:38:32.814ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85222025-04-30T06:38:32.822ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85232025-04-30T06:38:32.822ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85242025-04-30T06:38:32.822ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
85252025-04-30T06:38:32.822ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85262025-04-30T06:38:32.824ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e665505-9a6a-49a6-9395-bb483a72cdc5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85272025-04-30T06:38:32.824ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
85282025-04-30T06:38:32.825ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fe69220-9d7b-472e-8ee1-0589dbfc0bc7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85292025-04-30T06:38:32.826ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
85302025-04-30T06:38:32.826ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85312025-04-30T06:38:32.826ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
85322025-04-30T06:38:32.827ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85332025-04-30T06:38:32.828ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42924030-dc65-476e-939c-9d06e979aaa6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85342025-04-30T06:38:32.828ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
85352025-04-30T06:38:32.831ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be8e2669-ec95-4211-a952-55088f11e6b7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85362025-04-30T06:38:32.831ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
85372025-04-30T06:38:32.834ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f43b533-ec65-4f93-9e6f-f8d659e03f9c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85382025-04-30T06:38:32.834ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
85392025-04-30T06:38:32.836ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60156cf1-14f3-4871-b0de-b0c3a756f491", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85402025-04-30T06:38:32.836ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
85412025-04-30T06:38:32.840ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "760a404b-ec9f-474c-8173-aabe0e5e7bdf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85422025-04-30T06:38:32.840ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
85432025-04-30T06:38:32.842ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "413a0cb6-0464-47af-99e3-5e9a73503a37", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85442025-04-30T06:38:32.843ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
85452025-04-30T06:38:32.846ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "583e5b29-6b50-47bc-8460-1e9fd49abab6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85462025-04-30T06:38:32.846ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
85472025-04-30T06:38:32.847ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe9d9ecf-13a5-4501-bb7b-49b81854254e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85482025-04-30T06:38:32.847ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
85492025-04-30T06:38:32.849ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0b39125-d552-4ce4-9e70-1a1018986c95", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85502025-04-30T06:38:32.849ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
85512025-04-30T06:38:32.850ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85522025-04-30T06:38:32.850ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85532025-04-30T06:38:32.852ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f347e29-e945-49ed-866e-af6faa5d929b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85542025-04-30T06:38:32.852ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
85552025-04-30T06:38:32.853ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85562025-04-30T06:38:32.854ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85572025-04-30T06:38:32.854ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85582025-04-30T06:38:32.854ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85592025-04-30T06:38:32.855ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "094f8ede-74d6-4500-9e3c-a5e319db4aaf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85602025-04-30T06:38:32.855ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
85612025-04-30T06:38:32.856ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8078df0b-7669-48c9-8578-16cad3dbdd4a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85622025-04-30T06:38:32.857ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85632025-04-30T06:38:32.857ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85642025-04-30T06:38:32.857ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85652025-04-30T06:38:32.857ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85662025-04-30T06:38:32.858ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85672025-04-30T06:38:32.861ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
85682025-04-30T06:38:32.865ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85692025-04-30T06:38:32.878ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85702025-04-30T06:38:32.878ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85712025-04-30T06:38:32.878ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85722025-04-30T06:38:32.878ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85732025-04-30T06:38:32.878ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85742025-04-30T06:38:32.879ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85752025-04-30T06:38:32.907ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85762025-04-30T06:38:32.918ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85772025-04-30T06:38:32.918ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85782025-04-30T06:38:32.918ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85792025-04-30T06:38:32.919ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85802025-04-30T06:38:32.921ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85812025-04-30T06:38:32.921ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85822025-04-30T06:38:32.921ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85832025-04-30T06:38:32.921ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
85842025-04-30T06:38:32.922ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85852025-04-30T06:38:32.989ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
85862025-04-30T06:38:32.989ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
85872025-04-30T06:38:32.990ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85882025-04-30T06:38:32.998ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85892025-04-30T06:38:32.998ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85902025-04-30T06:38:32.998ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
85912025-04-30T06:38:32.998ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85922025-04-30T06:38:32.999ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:57821 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
85932025-04-30T06:38:33.005ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:57821 req_id = c8f9a7f8-d666-4a95-bd55-9b5a4fb7b51e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
85942025-04-30T06:38:33.027ZINFOtest_quotas (dropshot): request completed latency_us = 21709 local_addr = [::1]:55655 method = GET remote_addr = [::1]:57821 req_id = c8f9a7f8-d666-4a95-bd55-9b5a4fb7b51e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
85952025-04-30T06:38:33.028ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8f9a7f8-d666-4a95-bd55-9b5a4fb7b51e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
85962025-04-30T06:38:33.031ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
85972025-04-30T06:38:33.031ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
85982025-04-30T06:38:33.031ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
85992025-04-30T06:38:33.034ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86002025-04-30T06:38:33.034ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86012025-04-30T06:38:33.034ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86022025-04-30T06:38:33.034ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86032025-04-30T06:38:33.035ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86042025-04-30T06:38:33.036ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86052025-04-30T06:38:33.036ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
86062025-04-30T06:38:33.037ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86072025-04-30T06:38:33.048ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86082025-04-30T06:38:33.054ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86092025-04-30T06:38:33.054ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86102025-04-30T06:38:33.054ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86112025-04-30T06:38:33.055ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86122025-04-30T06:38:33.102ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
86132025-04-30T06:38:33.106ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86142025-04-30T06:38:33.106ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86152025-04-30T06:38:33.106ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
86162025-04-30T06:38:33.109ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
86172025-04-30T06:38:33.113ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
86182025-04-30T06:38:33.113ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
86192025-04-30T06:38:33.113ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 551.003699ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
86202025-04-30T06:38:33.113ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
86212025-04-30T06:38:33.113ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86222025-04-30T06:38:33.113ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86232025-04-30T06:38:33.113ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86242025-04-30T06:38:33.113ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86252025-04-30T06:38:33.115ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86262025-04-30T06:38:33.118ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
86272025-04-30T06:38:33.118ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
86282025-04-30T06:38:33.119ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86292025-04-30T06:38:33.124ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86302025-04-30T06:38:33.124ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86312025-04-30T06:38:33.124ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86322025-04-30T06:38:33.125ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86332025-04-30T06:38:33.131ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
86342025-04-30T06:38:33.131ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
86352025-04-30T06:38:33.131ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
86362025-04-30T06:38:33.131ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86372025-04-30T06:38:33.132ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ca ff 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51245 req_id = 1d635b08-ddde-4728-88c2-800e901cdb01
86382025-04-30T06:38:33.132ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51967,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51245 req_id = 1d635b08-ddde-4728-88c2-800e901cdb01
86392025-04-30T06:38:33.133ZDEBGtest_quotas (store): query key key = _mgs._tcp
86402025-04-30T06:38:33.133ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86412025-04-30T06:38:33.133ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51245 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1d635b08-ddde-4728-88c2-800e901cdb01
86422025-04-30T06:38:33.134ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3977, tv_nsec: 932993706 } })
86432025-04-30T06:38:33.134ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
86442025-04-30T06:38:33.134ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e0 ea 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61387 req_id = 1abcbc09-8a61-45c7-afc8-42be58a649f8
86452025-04-30T06:38:33.134ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57578,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61387 req_id = 1abcbc09-8a61-45c7-afc8-42be58a649f8
86462025-04-30T06:38:33.134ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
86472025-04-30T06:38:33.134ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61387 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1abcbc09-8a61-45c7-afc8-42be58a649f8
86482025-04-30T06:38:33.185ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
86492025-04-30T06:38:33.185ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86502025-04-30T06:38:33.185ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86512025-04-30T06:38:33.185ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86522025-04-30T06:38:33.185ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
86532025-04-30T06:38:33.186ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86542025-04-30T06:38:33.186ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ee 30 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .0..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57980 req_id = 6b540a19-fcb6-4859-922b-2b0346abd449
86552025-04-30T06:38:33.186ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60976,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57980 req_id = 6b540a19-fcb6-4859-922b-2b0346abd449
86562025-04-30T06:38:33.186ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
86572025-04-30T06:38:33.186ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60976, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57980 req_id = 6b540a19-fcb6-4859-922b-2b0346abd449
86582025-04-30T06:38:33.187ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
86592025-04-30T06:38:33.187ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
86602025-04-30T06:38:33.187ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
86612025-04-30T06:38:33.192ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86622025-04-30T06:38:33.200ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86632025-04-30T06:38:33.200ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86642025-04-30T06:38:33.200ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
86652025-04-30T06:38:33.201ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86662025-04-30T06:38:33.210ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bd334f9-c8ef-4e7e-a229-246596f74972", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86672025-04-30T06:38:33.210ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
86682025-04-30T06:38:33.211ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
86692025-04-30T06:38:33.213ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86702025-04-30T06:38:33.213ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
86712025-04-30T06:38:33.213ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86722025-04-30T06:38:33.214ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1286f2df-d3ce-4e7c-9681-5e743d5c7f9b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86732025-04-30T06:38:33.214ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
86742025-04-30T06:38:33.217ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e7c1481-1f22-456e-ab4d-58a1ecbcb7b3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86752025-04-30T06:38:33.217ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
86762025-04-30T06:38:33.228ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "795c1a99-e527-45f1-9d19-28c866a037f4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86772025-04-30T06:38:33.228ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
86782025-04-30T06:38:33.240ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "893dd01c-cc40-4a17-8cff-a131914f1059", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86792025-04-30T06:38:33.240ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
86802025-04-30T06:38:33.245ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02d379b3-b282-4d9a-9267-a664400e81bc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86812025-04-30T06:38:33.245ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
86822025-04-30T06:38:33.251ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "281ec9d5-4373-480e-9887-46f18a0fa762", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86832025-04-30T06:38:33.251ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
86842025-04-30T06:38:33.254ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa48d59a-1c1f-4f67-b219-c4961efddb8a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86852025-04-30T06:38:33.254ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
86862025-04-30T06:38:33.255ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcd9d438-378a-49b5-9f26-e16d98e4b97f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86872025-04-30T06:38:33.255ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
86882025-04-30T06:38:33.257ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a137b5bc-bb70-4652-b514-2ca1d39454b0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86892025-04-30T06:38:33.257ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
86902025-04-30T06:38:33.260ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab5c4e59-e5a5-4d5b-8ab2-a383587f6cc7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86912025-04-30T06:38:33.261ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
86922025-04-30T06:38:33.262ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bd825a5-c935-49e4-98fa-69542aa4e17c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86932025-04-30T06:38:33.263ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
86942025-04-30T06:38:33.265ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53fd4cb3-6889-4ef1-9cfa-93646e99ca30", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86952025-04-30T06:38:33.265ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
86962025-04-30T06:38:33.267ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "945256b0-50db-4107-82bf-8645a4893c82", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
86972025-04-30T06:38:33.274ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
86982025-04-30T06:38:33.275ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
86992025-04-30T06:38:33.279ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82364c6d-13a3-4d39-9bf4-09c6adf97bf7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87002025-04-30T06:38:33.279ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
87012025-04-30T06:38:33.282ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e3f9160-6a3a-48ed-ba06-e6cbefbaf2c9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87022025-04-30T06:38:33.282ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
87032025-04-30T06:38:33.284ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "630c91a6-63b2-4f40-a57f-fec478646bea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87042025-04-30T06:38:33.284ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
87052025-04-30T06:38:33.287ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a69eb05d-09a6-4695-aadb-921a7750e24f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87062025-04-30T06:38:33.287ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
87072025-04-30T06:38:33.290ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9d6bb08-4350-4eaa-aac2-52e30ed8a8aa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87082025-04-30T06:38:33.290ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
87092025-04-30T06:38:33.291ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dabc9504-7576-4328-b29f-e98a19da8b8e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87102025-04-30T06:38:33.291ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
87112025-04-30T06:38:33.292ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5bb2422-7d09-44a8-8625-466e7c7c8edb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87122025-04-30T06:38:33.292ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
87132025-04-30T06:38:33.293ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46442089-b070-497d-9c8b-acfc2112b583", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87142025-04-30T06:38:33.293ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
87152025-04-30T06:38:33.295ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "288c936d-4ad5-4fff-9e1d-746912aca17b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87162025-04-30T06:38:33.295ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
87172025-04-30T06:38:33.296ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "397c8c18-49b5-41bf-b15a-91025b821eab", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87182025-04-30T06:38:33.296ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
87192025-04-30T06:38:33.299ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "669a12d9-8fe7-444a-905f-6eb542606dc8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87202025-04-30T06:38:33.299ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
87212025-04-30T06:38:33.302ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77ef6286-66e7-4fb5-9449-8931fc868aa9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87222025-04-30T06:38:33.302ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
87232025-04-30T06:38:33.313ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b65e5c3-ae40-49c7-82f7-45ed4d2fdf22", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87242025-04-30T06:38:33.313ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
87252025-04-30T06:38:33.318ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69ca0906-4f8b-4d1c-8804-baab5a2aa22f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87262025-04-30T06:38:33.319ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
87272025-04-30T06:38:33.322ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf4cc51a-5544-4666-bdac-1fa10157f60a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87282025-04-30T06:38:33.322ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
87292025-04-30T06:38:33.324ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98b45d41-a008-4ed0-9629-bc1bcc9bb4a0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87302025-04-30T06:38:33.325ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
87312025-04-30T06:38:33.326ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87322025-04-30T06:38:33.326ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87332025-04-30T06:38:33.327ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87342025-04-30T06:38:33.327ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
87352025-04-30T06:38:33.328ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87362025-04-30T06:38:33.329ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "021046e6-87ad-4199-a241-58c44fa9be3b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87372025-04-30T06:38:33.330ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
87382025-04-30T06:38:33.332ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1806986-097c-4405-89b4-dc3eaeec33fc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87392025-04-30T06:38:33.332ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
87402025-04-30T06:38:33.335ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d2b90b6-635e-45a7-be6f-5d2cd705f5ac", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87412025-04-30T06:38:33.336ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
87422025-04-30T06:38:33.337ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8550688a-3583-4d0b-b9f4-38da7338187b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87432025-04-30T06:38:33.337ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
87442025-04-30T06:38:33.341ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87452025-04-30T06:38:33.347ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87462025-04-30T06:38:33.347ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87472025-04-30T06:38:33.347ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
87482025-04-30T06:38:33.348ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87492025-04-30T06:38:33.348ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d000481-77ab-4ddb-9016-d91552eaa2b8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87502025-04-30T06:38:33.348ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
87512025-04-30T06:38:33.351ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87522025-04-30T06:38:33.351ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
87532025-04-30T06:38:33.352ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87542025-04-30T06:38:33.358ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3123b053-5bda-4785-b4d6-f8ed419ecd36", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87552025-04-30T06:38:33.358ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
87562025-04-30T06:38:33.368ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1be93e32-a58b-432e-a7ac-f28c62f123d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87572025-04-30T06:38:33.368ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
87582025-04-30T06:38:33.381ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9762ba95-bcbf-4954-b5a9-b10c3c943e2e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87592025-04-30T06:38:33.381ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
87602025-04-30T06:38:33.384ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6465683f-a43f-414f-845d-d184c1285d75", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87612025-04-30T06:38:33.384ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
87622025-04-30T06:38:33.388ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7582968d-bf22-4b8c-b862-187a4604a2b8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87632025-04-30T06:38:33.389ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
87642025-04-30T06:38:33.391ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12c798ec-ca6e-4c6f-8486-c16c9dc0f1b0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87652025-04-30T06:38:33.391ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
87662025-04-30T06:38:33.399ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c427243f-a9a3-4881-aa2e-457f06ab7972", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87672025-04-30T06:38:33.400ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
87682025-04-30T06:38:33.413ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2b49167-dd86-4438-b609-1162d8ce11e0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87692025-04-30T06:38:33.413ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
87702025-04-30T06:38:33.415ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4b7b35d-fb6d-4de3-8a0b-f49342d16d91", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87712025-04-30T06:38:33.416ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
87722025-04-30T06:38:33.416ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
87732025-04-30T06:38:33.418ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ee87036-a6ab-41bb-902d-b8f6c5a6102d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
87742025-04-30T06:38:33.419ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
87752025-04-30T06:38:33.419ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87762025-04-30T06:38:33.419ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87772025-04-30T06:38:33.419ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87782025-04-30T06:38:33.421ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87792025-04-30T06:38:33.454ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87802025-04-30T06:38:33.463ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
87812025-04-30T06:38:33.463ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
87822025-04-30T06:38:33.463ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87832025-04-30T06:38:33.463ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87842025-04-30T06:38:33.463ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87852025-04-30T06:38:33.464ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87862025-04-30T06:38:33.470ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87872025-04-30T06:38:33.470ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87882025-04-30T06:38:33.470ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87892025-04-30T06:38:33.470ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
87902025-04-30T06:38:33.471ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87912025-04-30T06:38:33.479ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87922025-04-30T06:38:33.490ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
87932025-04-30T06:38:33.490ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87942025-04-30T06:38:33.490ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87952025-04-30T06:38:33.491ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87962025-04-30T06:38:33.496ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87972025-04-30T06:38:33.507ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87982025-04-30T06:38:33.507ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87992025-04-30T06:38:33.507ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
88002025-04-30T06:38:33.508ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88012025-04-30T06:38:33.595ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
88022025-04-30T06:38:33.595ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
88032025-04-30T06:38:33.597ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:41771 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
88042025-04-30T06:38:33.598ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:41771 req_id = 792ea091-1fa9-403d-a683-50b457a2dd3a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88052025-04-30T06:38:33.599ZINFOtest_quotas (dropshot): request completed latency_us = 776 local_addr = [::1]:55655 method = GET remote_addr = [::1]:41771 req_id = 792ea091-1fa9-403d-a683-50b457a2dd3a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88062025-04-30T06:38:33.599ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88072025-04-30T06:38:33.599ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
88082025-04-30T06:38:33.600ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88092025-04-30T06:38:33.600ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "792ea091-1fa9-403d-a683-50b457a2dd3a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:32 GMT"} })
88102025-04-30T06:38:33.604ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88112025-04-30T06:38:33.606ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88122025-04-30T06:38:33.606ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
88132025-04-30T06:38:33.610ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88142025-04-30T06:38:33.610ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88152025-04-30T06:38:33.610ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88162025-04-30T06:38:33.610ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88172025-04-30T06:38:33.611ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88182025-04-30T06:38:33.615ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88192025-04-30T06:38:33.628ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88202025-04-30T06:38:33.628ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88212025-04-30T06:38:33.628ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88222025-04-30T06:38:33.629ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88232025-04-30T06:38:33.659ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
88242025-04-30T06:38:33.663ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
88252025-04-30T06:38:33.663ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
88262025-04-30T06:38:33.663ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 550.234647ms iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
88272025-04-30T06:38:33.664ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
88282025-04-30T06:38:33.664ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88292025-04-30T06:38:33.664ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88302025-04-30T06:38:33.664ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88312025-04-30T06:38:33.664ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88322025-04-30T06:38:33.665ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88332025-04-30T06:38:33.668ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88342025-04-30T06:38:33.673ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88352025-04-30T06:38:33.673ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88362025-04-30T06:38:33.673ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88372025-04-30T06:38:33.673ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88382025-04-30T06:38:33.677ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
88392025-04-30T06:38:33.677ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
88402025-04-30T06:38:33.678ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 90 30 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .0..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49702 req_id = ca418de3-db1e-48c5-ace5-4c9f8ad91bc5
88412025-04-30T06:38:33.678ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36912,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49702 req_id = ca418de3-db1e-48c5-ace5-4c9f8ad91bc5
88422025-04-30T06:38:33.679ZDEBGtest_quotas (store): query key key = _mgs._tcp
88432025-04-30T06:38:33.679ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88442025-04-30T06:38:33.679ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49702 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ca418de3-db1e-48c5-ace5-4c9f8ad91bc5
88452025-04-30T06:38:33.680ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3978, tv_nsec: 479325366 } })
88462025-04-30T06:38:33.680ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
88472025-04-30T06:38:33.681ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 75 dc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e u...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57571 req_id = 6bd40c40-955d-4b64-b3f9-8029398bda8c
88482025-04-30T06:38:33.681ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30172,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57571 req_id = 6bd40c40-955d-4b64-b3f9-8029398bda8c
88492025-04-30T06:38:33.681ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88502025-04-30T06:38:33.681ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57571 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6bd40c40-955d-4b64-b3f9-8029398bda8c
88512025-04-30T06:38:33.753ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
88522025-04-30T06:38:33.753ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88532025-04-30T06:38:33.753ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88542025-04-30T06:38:33.754ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88552025-04-30T06:38:33.754ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
88562025-04-30T06:38:33.754ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88572025-04-30T06:38:33.754ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
88582025-04-30T06:38:33.755ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88592025-04-30T06:38:33.755ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = e3fb1ed6-d31a-453f-aebb-5117d1552077 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88602025-04-30T06:38:33.755ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e9 87 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63378 req_id = 47359dd0-50bf-4912-96fe-fc9052ad4ac6
88612025-04-30T06:38:33.756ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59783,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63378 req_id = 47359dd0-50bf-4912-96fe-fc9052ad4ac6
88622025-04-30T06:38:33.756ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
88632025-04-30T06:38:33.756ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59783, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63378 req_id = 47359dd0-50bf-4912-96fe-fc9052ad4ac6
88642025-04-30T06:38:33.756ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = e3fb1ed6-d31a-453f-aebb-5117d1552077 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88652025-04-30T06:38:33.756ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = e3fb1ed6-d31a-453f-aebb-5117d1552077 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88662025-04-30T06:38:33.757ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = e3fb1ed6-d31a-453f-aebb-5117d1552077 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88672025-04-30T06:38:33.757ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
88682025-04-30T06:38:33.758ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
88692025-04-30T06:38:33.758ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
88702025-04-30T06:38:33.759ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88712025-04-30T06:38:33.768ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88722025-04-30T06:38:33.768ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88732025-04-30T06:38:33.768ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
88742025-04-30T06:38:33.769ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88752025-04-30T06:38:33.771ZINFOtest_quotas (dropshot_internal): request completed latency_us = 15917 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = e3fb1ed6-d31a-453f-aebb-5117d1552077 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88762025-04-30T06:38:33.773ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3fb1ed6-d31a-453f-aebb-5117d1552077", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
88772025-04-30T06:38:33.774ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88782025-04-30T06:38:33.774ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88792025-04-30T06:38:33.774ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
88802025-04-30T06:38:33.775ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88812025-04-30T06:38:33.776ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = dfcf955f-efb2-4ff4-aad0-57fe4f1c1dc8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88822025-04-30T06:38:33.777ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = dfcf955f-efb2-4ff4-aad0-57fe4f1c1dc8 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88832025-04-30T06:38:33.777ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = dfcf955f-efb2-4ff4-aad0-57fe4f1c1dc8 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88842025-04-30T06:38:33.778ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = dfcf955f-efb2-4ff4-aad0-57fe4f1c1dc8 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88852025-04-30T06:38:33.778ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddea4f1e-5791-45a5-a7a6-470a8e14650c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
88862025-04-30T06:38:33.779ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
88872025-04-30T06:38:33.783ZINFOtest_quotas (dropshot_internal): request completed latency_us = 7526 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = dfcf955f-efb2-4ff4-aad0-57fe4f1c1dc8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
88882025-04-30T06:38:33.785ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfcf955f-efb2-4ff4-aad0-57fe4f1c1dc8", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
88892025-04-30T06:38:33.785ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
88902025-04-30T06:38:33.785ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88912025-04-30T06:38:33.785ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
88922025-04-30T06:38:33.785ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4692af1f-f33b-4eea-8d5f-427a2d27e828", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
88932025-04-30T06:38:33.785ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
88942025-04-30T06:38:33.790ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15e2cbe9-bdaa-4bee-b2c3-e6de305e096a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
88952025-04-30T06:38:33.791ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
88962025-04-30T06:38:33.795ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77736115-245c-4f4c-b1e4-94a2db0c0f3d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
88972025-04-30T06:38:33.795ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
88982025-04-30T06:38:33.801ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "662463cb-76bc-4896-ab3b-07498ba7e61d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
88992025-04-30T06:38:33.802ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
89002025-04-30T06:38:33.805ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1287c09-6a46-4674-bbdd-dd81a7e138a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89012025-04-30T06:38:33.806ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
89022025-04-30T06:38:33.810ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4920f24b-372d-4a0e-a4f6-d4766350b14f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89032025-04-30T06:38:33.810ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
89042025-04-30T06:38:33.815ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
89052025-04-30T06:38:33.817ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bea2487-d825-4815-9f5e-c2ade7c3d3c3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89062025-04-30T06:38:33.817ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
89072025-04-30T06:38:33.818ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "618c5072-dfa5-4402-aa31-eefc23f7ac87", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89082025-04-30T06:38:33.818ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
89092025-04-30T06:38:33.819ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2053827c-ac3d-48bc-bd43-3a28d71b7f83", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89102025-04-30T06:38:33.819ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
89112025-04-30T06:38:33.820ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5484b075-74a3-469c-91ee-9aaf4a9c6a37", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89122025-04-30T06:38:33.820ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
89132025-04-30T06:38:33.821ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f475acbf-b4bf-4517-883a-51b921b48e7c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89142025-04-30T06:38:33.821ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
89152025-04-30T06:38:33.823ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44ffe55b-c20e-47ea-b0f0-c6b70abd9c35", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89162025-04-30T06:38:33.823ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
89172025-04-30T06:38:33.825ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1c479e1-593f-4332-8907-1b2610af35e2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89182025-04-30T06:38:33.825ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
89192025-04-30T06:38:33.829ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d517910-336c-4c2b-a1f0-9e9f7900e2de", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89202025-04-30T06:38:33.829ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
89212025-04-30T06:38:33.830ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e578c46e-0f69-463a-99d3-3bad2c335268", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89222025-04-30T06:38:33.830ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
89232025-04-30T06:38:33.835ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2772959c-6402-4f75-a613-c91226c41e3d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89242025-04-30T06:38:33.835ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
89252025-04-30T06:38:33.838ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57b9daaf-1da1-4687-aa00-8ea40582dabc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89262025-04-30T06:38:33.838ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
89272025-04-30T06:38:33.841ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d5a7cd1-be8f-43a9-957a-382ef205f8dc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89282025-04-30T06:38:33.842ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
89292025-04-30T06:38:33.843ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "148ac81e-ccdd-40e9-a359-93c6fdff290d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89302025-04-30T06:38:33.844ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
89312025-04-30T06:38:33.845ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95836602-23fb-4ae3-8989-fe25a7b2fca6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89322025-04-30T06:38:33.845ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
89332025-04-30T06:38:33.847ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ddadb4c-eeb9-49e7-856d-a750c8b63129", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89342025-04-30T06:38:33.847ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
89352025-04-30T06:38:33.848ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e9bbd57-7155-4540-96da-62f8bafda2e6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89362025-04-30T06:38:33.849ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
89372025-04-30T06:38:33.850ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89382025-04-30T06:38:33.850ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e19c4f3-42fe-4af3-a911-fd767e581f85", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89392025-04-30T06:38:33.851ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
89402025-04-30T06:38:33.851ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89412025-04-30T06:38:33.854ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89422025-04-30T06:38:33.855ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89432025-04-30T06:38:33.855ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89442025-04-30T06:38:33.855ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89452025-04-30T06:38:33.858ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9992dde-7f87-4c33-a8d2-e7b3c9a85e19", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89462025-04-30T06:38:33.858ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
89472025-04-30T06:38:33.861ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52696312-73f0-4fb5-9d05-0e8bd4b523cf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89482025-04-30T06:38:33.862ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
89492025-04-30T06:38:33.864ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1b83f89-ab5d-47e1-af69-c8c957194db0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89502025-04-30T06:38:33.864ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
89512025-04-30T06:38:33.866ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89522025-04-30T06:38:33.866ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89532025-04-30T06:38:33.866ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89542025-04-30T06:38:33.866ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
89552025-04-30T06:38:33.868ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89562025-04-30T06:38:33.870ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3729cb09-8279-402c-b6ca-b40e7bde8940", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89572025-04-30T06:38:33.870ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
89582025-04-30T06:38:33.871ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89592025-04-30T06:38:33.880ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89602025-04-30T06:38:33.880ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89612025-04-30T06:38:33.880ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
89622025-04-30T06:38:33.881ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89632025-04-30T06:38:33.883ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89642025-04-30T06:38:33.883ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
89652025-04-30T06:38:33.884ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89662025-04-30T06:38:33.884ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e827cdab-8796-477e-acac-3d6a56da3d2a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89672025-04-30T06:38:33.885ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
89682025-04-30T06:38:33.890ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4910d1bf-1b15-424d-b0fb-1d38771be277", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89692025-04-30T06:38:33.890ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
89702025-04-30T06:38:33.892ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2059fd28-aa85-480b-a557-ddeb6794b37f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89712025-04-30T06:38:33.892ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
89722025-04-30T06:38:33.896ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30c4814f-efdc-44d6-8f67-63816685e800", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89732025-04-30T06:38:33.896ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
89742025-04-30T06:38:33.899ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f23d0528-c347-4114-b272-7ac97953c804", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89752025-04-30T06:38:33.899ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
89762025-04-30T06:38:33.901ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b08c943-8b84-4034-8a71-25e623325343", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89772025-04-30T06:38:33.901ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
89782025-04-30T06:38:33.904ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff5c8d8a-181a-4e9a-93c8-c7aaf1d54542", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89792025-04-30T06:38:33.904ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
89802025-04-30T06:38:33.907ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bea4c09-d6c1-4ba7-a4d8-de0ff26309b8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89812025-04-30T06:38:33.907ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
89822025-04-30T06:38:33.910ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3221d7e8-815c-4b39-887d-fe5c29718ebc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89832025-04-30T06:38:33.910ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
89842025-04-30T06:38:33.912ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2b66030-80ad-456d-81ba-6d155f95a08a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89852025-04-30T06:38:33.912ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
89862025-04-30T06:38:33.918ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e687f5b1-43d3-4564-b916-cc39e1ec04d0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89872025-04-30T06:38:33.919ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
89882025-04-30T06:38:33.919ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
89892025-04-30T06:38:33.969ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c922d579-9f83-4356-becb-133aff2a4305", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
89902025-04-30T06:38:33.969ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
89912025-04-30T06:38:33.973ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89922025-04-30T06:38:33.973ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89932025-04-30T06:38:33.973ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89942025-04-30T06:38:33.973ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
89952025-04-30T06:38:33.973ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89962025-04-30T06:38:33.982ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89972025-04-30T06:38:33.990ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89982025-04-30T06:38:33.990ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89992025-04-30T06:38:33.990ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
90002025-04-30T06:38:33.990ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90012025-04-30T06:38:33.993ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90022025-04-30T06:38:34.001ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
90032025-04-30T06:38:34.004ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90042025-04-30T06:38:34.007ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5e61f0e-616f-40dd-8892-5ad9fe127a91", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
90052025-04-30T06:38:34.007ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
90062025-04-30T06:38:34.012ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8325180e-3abd-49ce-be1f-9adde4059911", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
90072025-04-30T06:38:34.012ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
90082025-04-30T06:38:34.013ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97139ce8-4988-4a19-82c5-587582374368", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
90092025-04-30T06:38:34.014ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
90102025-04-30T06:38:34.017ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15b5951a-9c76-43db-a85d-e8d36670e49e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
90112025-04-30T06:38:34.017ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
90122025-04-30T06:38:34.019ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60b8f062-1de9-4956-a155-df8c7681a80e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
90132025-04-30T06:38:34.019ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90142025-04-30T06:38:34.019ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90152025-04-30T06:38:34.020ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90162025-04-30T06:38:34.021ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90172025-04-30T06:38:34.021ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90182025-04-30T06:38:34.025ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90192025-04-30T06:38:34.036ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90202025-04-30T06:38:34.036ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90212025-04-30T06:38:34.037ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90222025-04-30T06:38:34.037ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90232025-04-30T06:38:34.037ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90242025-04-30T06:38:34.038ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90252025-04-30T06:38:34.042ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90262025-04-30T06:38:34.048ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90272025-04-30T06:38:34.048ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90282025-04-30T06:38:34.048ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90292025-04-30T06:38:34.049ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90302025-04-30T06:38:34.109ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90312025-04-30T06:38:34.109ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90322025-04-30T06:38:34.110ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
90332025-04-30T06:38:34.111ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
90342025-04-30T06:38:34.124ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
90352025-04-30T06:38:34.128ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
90362025-04-30T06:38:34.135ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
90372025-04-30T06:38:34.137ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
90382025-04-30T06:38:34.138ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90392025-04-30T06:38:34.189ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90402025-04-30T06:38:34.189ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90412025-04-30T06:38:34.189ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90422025-04-30T06:38:34.189ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
90432025-04-30T06:38:34.190ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90442025-04-30T06:38:34.194ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90452025-04-30T06:38:34.202ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90462025-04-30T06:38:34.202ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90472025-04-30T06:38:34.202ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
90482025-04-30T06:38:34.202ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90492025-04-30T06:38:34.292ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
90502025-04-30T06:38:34.292ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
90512025-04-30T06:38:34.292ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:44684 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
90522025-04-30T06:38:34.294ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
90532025-04-30T06:38:34.294ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90542025-04-30T06:38:34.294ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
90552025-04-30T06:38:34.295ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90562025-04-30T06:38:34.295ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:44684 req_id = eae60f0a-87bb-4818-8a96-1c773d019528 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90572025-04-30T06:38:34.296ZINFOtest_quotas (dropshot): request completed latency_us = 1338 local_addr = [::1]:55655 method = GET remote_addr = [::1]:44684 req_id = eae60f0a-87bb-4818-8a96-1c773d019528 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90582025-04-30T06:38:34.297ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "eae60f0a-87bb-4818-8a96-1c773d019528", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
90592025-04-30T06:38:34.300ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90602025-04-30T06:38:34.300ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90612025-04-30T06:38:34.300ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
90622025-04-30T06:38:34.303ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90632025-04-30T06:38:34.303ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90642025-04-30T06:38:34.304ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90652025-04-30T06:38:34.304ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90662025-04-30T06:38:34.305ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90672025-04-30T06:38:34.314ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90682025-04-30T06:38:34.321ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90692025-04-30T06:38:34.322ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90702025-04-30T06:38:34.322ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90712025-04-30T06:38:34.323ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90722025-04-30T06:38:34.373ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
90732025-04-30T06:38:34.374ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
90742025-04-30T06:38:34.374ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
90752025-04-30T06:38:34.374ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 710.913128ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
90762025-04-30T06:38:34.374ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
90772025-04-30T06:38:34.374ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90782025-04-30T06:38:34.374ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90792025-04-30T06:38:34.374ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90802025-04-30T06:38:34.375ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90812025-04-30T06:38:34.376ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90822025-04-30T06:38:34.378ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90832025-04-30T06:38:34.402ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90842025-04-30T06:38:34.402ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90852025-04-30T06:38:34.402ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90862025-04-30T06:38:34.404ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90872025-04-30T06:38:34.410ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
90882025-04-30T06:38:34.410ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
90892025-04-30T06:38:34.411ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b8 a0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52245 req_id = 767862c1-8555-45ce-904e-0f5e4c7863fb
90902025-04-30T06:38:34.411ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47264,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52245 req_id = 767862c1-8555-45ce-904e-0f5e4c7863fb
90912025-04-30T06:38:34.411ZDEBGtest_quotas (store): query key key = _mgs._tcp
90922025-04-30T06:38:34.412ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
90932025-04-30T06:38:34.412ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52245 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 767862c1-8555-45ce-904e-0f5e4c7863fb
90942025-04-30T06:38:34.412ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3979, tv_nsec: 211774483 } })
90952025-04-30T06:38:34.413ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
90962025-04-30T06:38:34.413ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 24 e1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e $...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56003 req_id = 0171a506-db2b-4de1-8e62-51054a2b5a6e
90972025-04-30T06:38:34.413ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9441,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56003 req_id = 0171a506-db2b-4de1-8e62-51054a2b5a6e
90982025-04-30T06:38:34.414ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
90992025-04-30T06:38:34.414ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56003 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0171a506-db2b-4de1-8e62-51054a2b5a6e
91002025-04-30T06:38:34.493ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
91012025-04-30T06:38:34.494ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91022025-04-30T06:38:34.494ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91032025-04-30T06:38:34.494ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91042025-04-30T06:38:34.494ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
91052025-04-30T06:38:34.495ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91062025-04-30T06:38:34.496ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 77 ab 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c w............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55163 req_id = ae49e2cd-c54d-43e1-8427-c08c0305405b
91072025-04-30T06:38:34.496ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30635,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55163 req_id = ae49e2cd-c54d-43e1-8427-c08c0305405b
91082025-04-30T06:38:34.496ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
91092025-04-30T06:38:34.497ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30635, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55163 req_id = ae49e2cd-c54d-43e1-8427-c08c0305405b
91102025-04-30T06:38:34.497ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
91112025-04-30T06:38:34.498ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
91122025-04-30T06:38:34.498ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
91132025-04-30T06:38:34.499ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91142025-04-30T06:38:34.508ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91152025-04-30T06:38:34.508ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91162025-04-30T06:38:34.508ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
91172025-04-30T06:38:34.508ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91182025-04-30T06:38:34.514ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91192025-04-30T06:38:34.514ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
91202025-04-30T06:38:34.514ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91212025-04-30T06:38:34.514ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0abf82f3-2b4f-4cc6-9b3a-7dca774fe7b9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91222025-04-30T06:38:34.515ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
91232025-04-30T06:38:34.517ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce780859-06e8-491c-b74b-e5e7b8ac1906", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91242025-04-30T06:38:34.517ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
91252025-04-30T06:38:34.536ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "774a3b40-621f-4485-94a7-b0624c2b9e7d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91262025-04-30T06:38:34.537ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
91272025-04-30T06:38:34.540ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a0a4302-2ee4-47f2-bde4-10765856841e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91282025-04-30T06:38:34.541ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
91292025-04-30T06:38:34.560ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43d32a90-5a8b-4b47-88f8-3e5f03c3faa6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91302025-04-30T06:38:34.560ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
91312025-04-30T06:38:34.566ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f54721ba-6aa6-4c58-8b37-0c590a6aa93d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91322025-04-30T06:38:34.566ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
91332025-04-30T06:38:34.574ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
91342025-04-30T06:38:34.582ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "672d46c5-d906-42be-90a4-a782ae7daa31", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91352025-04-30T06:38:34.582ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
91362025-04-30T06:38:34.586ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89b7b0d8-5646-44a5-bc3a-4617c171b32d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91372025-04-30T06:38:34.586ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
91382025-04-30T06:38:34.587ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28a3293b-03ea-445c-b683-71ff2b4711a7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91392025-04-30T06:38:34.587ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
91402025-04-30T06:38:34.588ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f1c16b8-8724-469b-9d08-42a751f2fba4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91412025-04-30T06:38:34.588ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
91422025-04-30T06:38:34.589ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d1b1db0-b987-408c-a751-e79819b1e5a6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91432025-04-30T06:38:34.590ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
91442025-04-30T06:38:34.591ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b15955bb-bb4e-4dbc-a013-35745bac2a22", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91452025-04-30T06:38:34.591ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
91462025-04-30T06:38:34.593ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcef9efe-e9aa-47e1-b23c-77fe83633988", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91472025-04-30T06:38:34.593ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
91482025-04-30T06:38:34.595ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6384306e-e5d1-4659-8330-833c508cffad", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91492025-04-30T06:38:34.595ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
91502025-04-30T06:38:34.601ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eedd8ed-4dcf-4fb7-87bf-147838430845", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91512025-04-30T06:38:34.602ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
91522025-04-30T06:38:34.604ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e2728e9-9bf8-4761-8d9a-0c8b2ba8a6fd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91532025-04-30T06:38:34.604ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
91542025-04-30T06:38:34.606ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae601004-ef46-4042-b1eb-4a00cb413341", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91552025-04-30T06:38:34.607ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
91562025-04-30T06:38:34.610ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb5d943e-0f89-4d0e-b950-056fe68f8497", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91572025-04-30T06:38:34.610ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
91582025-04-30T06:38:34.613ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e845e79a-9c5c-46d6-a2e9-1fdda6fe508e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91592025-04-30T06:38:34.613ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
91602025-04-30T06:38:34.614ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd11b286-be57-414d-9eb8-0634f25f03e1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91612025-04-30T06:38:34.614ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
91622025-04-30T06:38:34.616ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9064f6be-4be3-4d8d-a6e4-1cc9281fcbfd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91632025-04-30T06:38:34.616ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
91642025-04-30T06:38:34.619ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d36e4198-65c2-434e-9538-534824807ba0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91652025-04-30T06:38:34.619ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
91662025-04-30T06:38:34.620ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3e8f411-0f4e-4050-9fb1-d8f7756318e2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91672025-04-30T06:38:34.620ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
91682025-04-30T06:38:34.621ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d06f5b28-17bd-4932-ac6e-87f280f23cec", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91692025-04-30T06:38:34.621ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
91702025-04-30T06:38:34.623ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32e9c95b-5e8c-451d-8f4d-ee4c961a6408", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91712025-04-30T06:38:34.623ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
91722025-04-30T06:38:34.625ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91732025-04-30T06:38:34.625ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91742025-04-30T06:38:34.625ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91752025-04-30T06:38:34.625ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
91762025-04-30T06:38:34.626ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91772025-04-30T06:38:34.629ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfe17658-597c-4004-82a3-8b9a60b4ab0e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91782025-04-30T06:38:34.629ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
91792025-04-30T06:38:34.629ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91802025-04-30T06:38:34.637ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91812025-04-30T06:38:34.637ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91822025-04-30T06:38:34.637ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
91832025-04-30T06:38:34.637ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91842025-04-30T06:38:34.641ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91852025-04-30T06:38:34.641ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
91862025-04-30T06:38:34.644ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91872025-04-30T06:38:34.650ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e7798a3-9cda-4cec-ad30-c8b145311554", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91882025-04-30T06:38:34.650ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
91892025-04-30T06:38:34.653ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0f42ad9-cfe0-4ebe-a314-33fb91513317", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91902025-04-30T06:38:34.653ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
91912025-04-30T06:38:34.656ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b8acfb6-1c84-4893-ae39-1684a7365544", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91922025-04-30T06:38:34.656ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
91932025-04-30T06:38:34.670ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bd43d9b-6132-4d37-991b-0a30b358ffd5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91942025-04-30T06:38:34.670ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
91952025-04-30T06:38:34.671ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37320da2-4e71-4cd0-acce-b0c06a26110d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91962025-04-30T06:38:34.672ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
91972025-04-30T06:38:34.674ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a82e3ce-4738-4a55-99bb-53ec9b5674ea", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
91982025-04-30T06:38:34.674ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
91992025-04-30T06:38:34.677ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "080cb1e8-39b6-419e-87c0-e600482e3da5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92002025-04-30T06:38:34.678ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
92012025-04-30T06:38:34.680ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88643b2b-b5ba-49c9-ab19-3959339a852e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92022025-04-30T06:38:34.680ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
92032025-04-30T06:38:34.683ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1987c40-f3a9-41bf-9977-e6136b5b6e7e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92042025-04-30T06:38:34.683ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
92052025-04-30T06:38:34.687ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8251f69b-d6aa-4a15-8432-67ee202a5151", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92062025-04-30T06:38:34.689ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
92072025-04-30T06:38:34.698ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66863d37-42a5-4ab4-b679-fab61d825de8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92082025-04-30T06:38:34.698ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
92092025-04-30T06:38:34.700ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
92102025-04-30T06:38:34.704ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71876c67-daa0-4f57-990c-6920f86ef793", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92112025-04-30T06:38:34.704ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
92122025-04-30T06:38:34.708ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f72e73e-690f-423f-b3de-e18c6078abb3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92132025-04-30T06:38:34.708ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
92142025-04-30T06:38:34.711ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e48f3540-99f5-41da-82ef-efc52f8ef3d2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92152025-04-30T06:38:34.712ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
92162025-04-30T06:38:34.714ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cef842ec-408e-46b5-bf86-19ea7975249f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92172025-04-30T06:38:34.714ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
92182025-04-30T06:38:34.715ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b42aeb7-9131-4aab-91ff-11d3389ea732", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92192025-04-30T06:38:34.715ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
92202025-04-30T06:38:34.721ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e52e2f9-6204-429f-a60e-900d2264a0a8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92212025-04-30T06:38:34.722ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
92222025-04-30T06:38:34.724ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "061ade21-b625-4165-a58b-65fda912134f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92232025-04-30T06:38:34.724ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
92242025-04-30T06:38:34.725ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01021498-3809-407d-9a3f-5fcec1ec93d5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:33 GMT"} })
92252025-04-30T06:38:34.725ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92262025-04-30T06:38:34.726ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92272025-04-30T06:38:34.726ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92282025-04-30T06:38:34.726ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92292025-04-30T06:38:34.727ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92302025-04-30T06:38:34.729ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92312025-04-30T06:38:34.733ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92322025-04-30T06:38:34.733ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92332025-04-30T06:38:34.733ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92342025-04-30T06:38:34.733ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92352025-04-30T06:38:34.733ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92362025-04-30T06:38:34.734ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92372025-04-30T06:38:34.737ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92382025-04-30T06:38:34.744ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92392025-04-30T06:38:34.744ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92402025-04-30T06:38:34.744ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92412025-04-30T06:38:34.746ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92422025-04-30T06:38:34.825ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
92432025-04-30T06:38:34.825ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
92442025-04-30T06:38:34.826ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92452025-04-30T06:38:34.826ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92462025-04-30T06:38:34.826ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92472025-04-30T06:38:34.826ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
92482025-04-30T06:38:34.827ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92492025-04-30T06:38:34.827ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:33268 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
92502025-04-30T06:38:34.828ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:33268 req_id = 7b5b8e82-1008-4f10-9699-7d056fdd78a4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92512025-04-30T06:38:34.829ZINFOtest_quotas (dropshot): request completed latency_us = 931 local_addr = [::1]:55655 method = GET remote_addr = [::1]:33268 req_id = 7b5b8e82-1008-4f10-9699-7d056fdd78a4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92522025-04-30T06:38:34.830ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b5b8e82-1008-4f10-9699-7d056fdd78a4", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
92532025-04-30T06:38:34.832ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92542025-04-30T06:38:34.832ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92552025-04-30T06:38:34.832ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
92562025-04-30T06:38:34.834ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92572025-04-30T06:38:34.834ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92582025-04-30T06:38:34.834ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92592025-04-30T06:38:34.834ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92602025-04-30T06:38:34.835ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92612025-04-30T06:38:34.836ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92622025-04-30T06:38:34.842ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92632025-04-30T06:38:34.842ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92642025-04-30T06:38:34.842ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
92652025-04-30T06:38:34.842ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92662025-04-30T06:38:34.845ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92672025-04-30T06:38:34.850ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92682025-04-30T06:38:34.851ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92692025-04-30T06:38:34.851ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92702025-04-30T06:38:34.851ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92712025-04-30T06:38:34.853ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92722025-04-30T06:38:34.853ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92732025-04-30T06:38:34.854ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92742025-04-30T06:38:34.855ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
92752025-04-30T06:38:34.855ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92762025-04-30T06:38:34.856ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92772025-04-30T06:38:34.857ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92782025-04-30T06:38:34.857ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92792025-04-30T06:38:34.857ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92802025-04-30T06:38:34.901ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
92812025-04-30T06:38:34.902ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
92822025-04-30T06:38:34.906ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 531.854305ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
92832025-04-30T06:38:34.907ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
92842025-04-30T06:38:34.907ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92852025-04-30T06:38:34.907ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92862025-04-30T06:38:34.907ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92872025-04-30T06:38:34.907ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92882025-04-30T06:38:34.907ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92892025-04-30T06:38:34.911ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92902025-04-30T06:38:34.916ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92912025-04-30T06:38:34.916ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92922025-04-30T06:38:34.916ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92932025-04-30T06:38:34.917ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92942025-04-30T06:38:34.918ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
92952025-04-30T06:38:34.925ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
92962025-04-30T06:38:34.925ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
92972025-04-30T06:38:34.933ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 78 83 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 x............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61368 req_id = b20b71a2-bff2-4984-af8e-b39f4fe3f0a9
92982025-04-30T06:38:34.933ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30851,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61368 req_id = b20b71a2-bff2-4984-af8e-b39f4fe3f0a9
92992025-04-30T06:38:34.933ZDEBGtest_quotas (store): query key key = _mgs._tcp
93002025-04-30T06:38:34.933ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93012025-04-30T06:38:34.934ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61368 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b20b71a2-bff2-4984-af8e-b39f4fe3f0a9
93022025-04-30T06:38:34.935ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3979, tv_nsec: 734127187 } })
93032025-04-30T06:38:34.935ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
93042025-04-30T06:38:34.945ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b0 21 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .!..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58226 req_id = e21f5731-8f30-4b9a-bade-e3f713ea9a17
93052025-04-30T06:38:34.945ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45089,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58226 req_id = e21f5731-8f30-4b9a-bade-e3f713ea9a17
93062025-04-30T06:38:34.945ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93072025-04-30T06:38:34.945ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58226 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e21f5731-8f30-4b9a-bade-e3f713ea9a17
93082025-04-30T06:38:34.994ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
93092025-04-30T06:38:34.996ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93102025-04-30T06:38:34.997ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93112025-04-30T06:38:34.997ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93122025-04-30T06:38:34.997ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
93132025-04-30T06:38:34.997ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93142025-04-30T06:38:35.002ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 98 48 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .H..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58573 req_id = 8c410d6d-4a55-40cc-b821-0090194c447f
93152025-04-30T06:38:35.002ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38984,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58573 req_id = 8c410d6d-4a55-40cc-b821-0090194c447f
93162025-04-30T06:38:35.002ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
93172025-04-30T06:38:35.002ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38984, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58573 req_id = 8c410d6d-4a55-40cc-b821-0090194c447f
93182025-04-30T06:38:35.003ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
93192025-04-30T06:38:35.003ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
93202025-04-30T06:38:35.003ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
93212025-04-30T06:38:35.005ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93222025-04-30T06:38:35.010ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93232025-04-30T06:38:35.010ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93242025-04-30T06:38:35.010ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
93252025-04-30T06:38:35.010ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93262025-04-30T06:38:35.019ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93272025-04-30T06:38:35.019ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
93282025-04-30T06:38:35.019ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93292025-04-30T06:38:35.020ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f446db2-a627-45db-8633-8330f1354e0c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93302025-04-30T06:38:35.021ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
93312025-04-30T06:38:35.039ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1831253b-2b8d-4133-9c82-ca5459d79ea6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93322025-04-30T06:38:35.039ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
93332025-04-30T06:38:35.041ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a3a550a-e0ed-4c9a-b9b4-83476e17d27b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93342025-04-30T06:38:35.041ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
93352025-04-30T06:38:35.043ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d3055dc-251f-4a51-b6d4-00b95add980d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93362025-04-30T06:38:35.043ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
93372025-04-30T06:38:35.048ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e5c1031-ad95-439c-bba3-e798383825cf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93382025-04-30T06:38:35.048ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
93392025-04-30T06:38:35.056ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da775d5d-43bc-4277-9013-29829c587f63", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93402025-04-30T06:38:35.057ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
93412025-04-30T06:38:35.062ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9da6884-4ac3-46c5-9866-bd9b3b1f7821", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93422025-04-30T06:38:35.062ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
93432025-04-30T06:38:35.064ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6fc6836-5507-4e66-9f27-27b2a567769f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93442025-04-30T06:38:35.064ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
93452025-04-30T06:38:35.066ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45cef210-3c36-4922-abdb-5755e6253c27", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93462025-04-30T06:38:35.066ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
93472025-04-30T06:38:35.067ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "754a94e1-9ae4-4b06-82fc-a91e486ef06f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93482025-04-30T06:38:35.068ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
93492025-04-30T06:38:35.069ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93057b69-2169-4a0e-8207-6bb4453a0a91", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93502025-04-30T06:38:35.069ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
93512025-04-30T06:38:35.072ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4d1bb73-e5e5-4d13-b06c-a6b0053ddc58", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93522025-04-30T06:38:35.072ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
93532025-04-30T06:38:35.073ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
93542025-04-30T06:38:35.074ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47526302-6832-4eb1-be10-1f1d42556bdc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93552025-04-30T06:38:35.074ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
93562025-04-30T06:38:35.076ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "caef2140-5a0a-4f20-b298-4d503b03965b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93572025-04-30T06:38:35.076ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
93582025-04-30T06:38:35.078ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aefd1c87-9689-447f-9685-6683b33bfe1f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93592025-04-30T06:38:35.078ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
93602025-04-30T06:38:35.081ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "545f5f6f-2367-483b-a45c-98155cea5bff", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93612025-04-30T06:38:35.081ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
93622025-04-30T06:38:35.082ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05c094e6-0ca8-4f44-97ed-9cf3b247a1e3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93632025-04-30T06:38:35.082ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
93642025-04-30T06:38:35.084ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ac1d51d-38f1-4805-96a8-1f7ceef1bf6b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93652025-04-30T06:38:35.084ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
93662025-04-30T06:38:35.088ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7db4949b-caf5-4790-97b6-5a0f70f8992e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93672025-04-30T06:38:35.088ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
93682025-04-30T06:38:35.089ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d93236d8-699c-4ff9-9475-c70419d8b9be", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93692025-04-30T06:38:35.089ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
93702025-04-30T06:38:35.090ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14ee961e-020f-45f2-856e-14987f2e68e0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93712025-04-30T06:38:35.090ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
93722025-04-30T06:38:35.092ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8b0fbc4-b121-4649-84ba-857d348edaae", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93732025-04-30T06:38:35.092ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
93742025-04-30T06:38:35.093ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "932a44c9-901e-4dcd-9010-bc628a4c461d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93752025-04-30T06:38:35.093ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
93762025-04-30T06:38:35.098ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e63f5eb1-874b-4c06-af64-2be1071b81c9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93772025-04-30T06:38:35.098ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
93782025-04-30T06:38:35.101ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e40bc2be-551b-4dd3-89ab-791849e17509", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93792025-04-30T06:38:35.101ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
93802025-04-30T06:38:35.104ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
93812025-04-30T06:38:35.104ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
93822025-04-30T06:38:35.104ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
93832025-04-30T06:38:35.105ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
93842025-04-30T06:38:35.106ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27cc4585-a068-4e85-923e-3ffb39ba1158", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93852025-04-30T06:38:35.106ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
93862025-04-30T06:38:35.112ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ba5234e-db86-4fa4-a67a-cbf1031a1075", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93872025-04-30T06:38:35.113ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
93882025-04-30T06:38:35.113ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
93892025-04-30T06:38:35.113ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
93902025-04-30T06:38:35.115ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9382a6c-c40c-4e79-ae56-e6fad67c9fb5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93912025-04-30T06:38:35.115ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
93922025-04-30T06:38:35.117ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
93932025-04-30T06:38:35.117ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
93942025-04-30T06:38:35.118ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56a78361-2ba1-4df3-9bd2-9ee2dfbf912f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93952025-04-30T06:38:35.118ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
93962025-04-30T06:38:35.121ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "148b6b8b-1812-4f17-bcc0-536228ba173f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93972025-04-30T06:38:35.121ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
93982025-04-30T06:38:35.122ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "154dd317-8213-4ffc-9188-d0a1faa239a4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
93992025-04-30T06:38:35.122ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
94002025-04-30T06:38:35.123ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "804180b7-ce47-4381-ad7d-19e36a384e36", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94012025-04-30T06:38:35.123ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
94022025-04-30T06:38:35.124ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94032025-04-30T06:38:35.124ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94042025-04-30T06:38:35.124ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94052025-04-30T06:38:35.124ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
94062025-04-30T06:38:35.125ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94072025-04-30T06:38:35.126ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "492474a5-168d-440c-9302-80c9e45e4fb0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94082025-04-30T06:38:35.126ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
94092025-04-30T06:38:35.128ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaf2554d-2762-4ec3-ab9c-53823b9e7893", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94102025-04-30T06:38:35.128ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
94112025-04-30T06:38:35.128ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94122025-04-30T06:38:35.140ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94132025-04-30T06:38:35.140ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94142025-04-30T06:38:35.141ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
94152025-04-30T06:38:35.141ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94162025-04-30T06:38:35.146ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61bf4f67-9998-42bc-a1d8-c849d1a7f502", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94172025-04-30T06:38:35.146ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
94182025-04-30T06:38:35.148ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94192025-04-30T06:38:35.148ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
94202025-04-30T06:38:35.148ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94212025-04-30T06:38:35.151ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3058f88-ae30-452b-b917-9ec98fb9ac84", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94222025-04-30T06:38:35.152ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
94232025-04-30T06:38:35.154ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c32e3ec-32f6-4513-826f-8e4f27fde123", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94242025-04-30T06:38:35.154ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
94252025-04-30T06:38:35.156ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "131fdd09-3b06-4530-96db-7de12148b733", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94262025-04-30T06:38:35.156ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
94272025-04-30T06:38:35.158ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f444f9e8-bc8e-4dc5-b376-46b7792f8f59", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94282025-04-30T06:38:35.159ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
94292025-04-30T06:38:35.161ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd827cae-d8de-4d0b-9f36-f1ddc1ef427a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94302025-04-30T06:38:35.161ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
94312025-04-30T06:38:35.163ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e68679e3-b69c-4bd1-8390-a13adab49099", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94322025-04-30T06:38:35.163ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
94332025-04-30T06:38:35.164ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc8730b8-eb5d-4c73-919f-459c992cc48a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94342025-04-30T06:38:35.164ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
94352025-04-30T06:38:35.165ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d323ff88-ee0e-481a-b1a8-805ac13f7696", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94362025-04-30T06:38:35.165ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
94372025-04-30T06:38:35.166ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "865bdc60-c88b-4016-9eae-e97e60bb0e9b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94382025-04-30T06:38:35.167ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
94392025-04-30T06:38:35.167ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6083f42c-f135-4093-829e-2c75cdba985b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94402025-04-30T06:38:35.168ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94412025-04-30T06:38:35.168ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94422025-04-30T06:38:35.168ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94432025-04-30T06:38:35.168ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94442025-04-30T06:38:35.168ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94452025-04-30T06:38:35.174ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94462025-04-30T06:38:35.179ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94472025-04-30T06:38:35.180ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94482025-04-30T06:38:35.181ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94492025-04-30T06:38:35.181ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94502025-04-30T06:38:35.181ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94512025-04-30T06:38:35.182ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94522025-04-30T06:38:35.185ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94532025-04-30T06:38:35.190ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94542025-04-30T06:38:35.190ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94552025-04-30T06:38:35.190ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94562025-04-30T06:38:35.191ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94572025-04-30T06:38:35.200ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
94582025-04-30T06:38:35.301ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
94592025-04-30T06:38:35.301ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
94602025-04-30T06:38:35.305ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94612025-04-30T06:38:35.307ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94622025-04-30T06:38:35.307ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94632025-04-30T06:38:35.307ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
94642025-04-30T06:38:35.308ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94652025-04-30T06:38:35.308ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:39285 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
94662025-04-30T06:38:35.309ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
94672025-04-30T06:38:35.322ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:39285 req_id = b1ba9e3c-b6b8-4d88-8622-92ad71e67c01 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94682025-04-30T06:38:35.324ZINFOtest_quotas (dropshot): request completed latency_us = 2153 local_addr = [::1]:55655 method = GET remote_addr = [::1]:39285 req_id = b1ba9e3c-b6b8-4d88-8622-92ad71e67c01 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94692025-04-30T06:38:35.326ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1ba9e3c-b6b8-4d88-8622-92ad71e67c01", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
94702025-04-30T06:38:35.328ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94712025-04-30T06:38:35.328ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
94722025-04-30T06:38:35.328ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
94732025-04-30T06:38:35.331ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
94742025-04-30T06:38:35.331ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94752025-04-30T06:38:35.333ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94762025-04-30T06:38:35.333ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94772025-04-30T06:38:35.334ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94782025-04-30T06:38:35.334ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94792025-04-30T06:38:35.345ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94802025-04-30T06:38:35.345ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94812025-04-30T06:38:35.345ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
94822025-04-30T06:38:35.346ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94832025-04-30T06:38:35.349ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94842025-04-30T06:38:35.367ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
94852025-04-30T06:38:35.367ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94862025-04-30T06:38:35.367ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94872025-04-30T06:38:35.368ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94882025-04-30T06:38:35.375ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94892025-04-30T06:38:35.375ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
94902025-04-30T06:38:35.376ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94912025-04-30T06:38:35.528ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
94922025-04-30T06:38:35.528ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
94932025-04-30T06:38:35.528ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 621.632512ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
94942025-04-30T06:38:35.528ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
94952025-04-30T06:38:35.529ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
94962025-04-30T06:38:35.529ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94972025-04-30T06:38:35.529ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94982025-04-30T06:38:35.529ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94992025-04-30T06:38:35.530ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95002025-04-30T06:38:35.537ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95012025-04-30T06:38:35.543ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95022025-04-30T06:38:35.544ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95032025-04-30T06:38:35.544ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95042025-04-30T06:38:35.544ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95052025-04-30T06:38:35.547ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
95062025-04-30T06:38:35.547ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
95072025-04-30T06:38:35.551ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6c 43 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 lC..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61747 req_id = 50a326ff-cf2c-40c1-8c10-5c58e47429fb
95082025-04-30T06:38:35.551ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27715,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61747 req_id = 50a326ff-cf2c-40c1-8c10-5c58e47429fb
95092025-04-30T06:38:35.551ZDEBGtest_quotas (store): query key key = _mgs._tcp
95102025-04-30T06:38:35.551ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95112025-04-30T06:38:35.551ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61747 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 50a326ff-cf2c-40c1-8c10-5c58e47429fb
95122025-04-30T06:38:35.551ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
95132025-04-30T06:38:35.552ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3980, tv_nsec: 351112791 } })
95142025-04-30T06:38:35.552ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
95152025-04-30T06:38:35.553ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 91 c2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49679 req_id = a04aa6aa-d13d-4113-9d54-2ffe3df9329c
95162025-04-30T06:38:35.553ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37314,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49679 req_id = a04aa6aa-d13d-4113-9d54-2ffe3df9329c
95172025-04-30T06:38:35.553ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95182025-04-30T06:38:35.553ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49679 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a04aa6aa-d13d-4113-9d54-2ffe3df9329c
95192025-04-30T06:38:35.623ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
95202025-04-30T06:38:35.624ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95212025-04-30T06:38:35.624ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95222025-04-30T06:38:35.624ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95232025-04-30T06:38:35.624ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
95242025-04-30T06:38:35.625ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95252025-04-30T06:38:35.627ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 25 5f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c %_..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54740 req_id = 726abf0e-a23d-4fdd-8a44-977ccb7bd025
95262025-04-30T06:38:35.628ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9567,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54740 req_id = 726abf0e-a23d-4fdd-8a44-977ccb7bd025
95272025-04-30T06:38:35.629ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
95282025-04-30T06:38:35.629ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9567, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54740 req_id = 726abf0e-a23d-4fdd-8a44-977ccb7bd025
95292025-04-30T06:38:35.629ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
95302025-04-30T06:38:35.629ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
95312025-04-30T06:38:35.629ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
95322025-04-30T06:38:35.630ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95332025-04-30T06:38:35.642ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95342025-04-30T06:38:35.642ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95352025-04-30T06:38:35.642ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
95362025-04-30T06:38:35.643ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95372025-04-30T06:38:35.655ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "cee36e5a-f39d-476f-9dd3-2cb1b86af0fd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95382025-04-30T06:38:35.655ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
95392025-04-30T06:38:35.656ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95402025-04-30T06:38:35.657ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
95412025-04-30T06:38:35.658ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95422025-04-30T06:38:35.661ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bc7b0e7-ab2f-4e9a-b5da-54985ff3fa6b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95432025-04-30T06:38:35.662ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
95442025-04-30T06:38:35.667ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71b496dc-f70d-4939-95e1-0030f211b329", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95452025-04-30T06:38:35.672ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
95462025-04-30T06:38:35.674ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88c399fe-4689-4efe-babf-fd06123be73f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95472025-04-30T06:38:35.674ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
95482025-04-30T06:38:35.677ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba9df238-7b2e-4186-bd6d-7c374b03c68a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95492025-04-30T06:38:35.677ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
95502025-04-30T06:38:35.742ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
95512025-04-30T06:38:35.742ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
95522025-04-30T06:38:35.743ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = a6af6d61-846e-4c70-978f-1998815a48e2 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
95532025-04-30T06:38:35.744ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = a6af6d61-846e-4c70-978f-1998815a48e2 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
95542025-04-30T06:38:35.744ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = a6af6d61-846e-4c70-978f-1998815a48e2 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
95552025-04-30T06:38:35.756ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = a6af6d61-846e-4c70-978f-1998815a48e2 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
95562025-04-30T06:38:35.791ZINFOtest_quotas (dropshot_internal): request completed latency_us = 48188 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = a6af6d61-846e-4c70-978f-1998815a48e2 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
95572025-04-30T06:38:35.806ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6af6d61-846e-4c70-978f-1998815a48e2", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95582025-04-30T06:38:35.807ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95592025-04-30T06:38:35.811ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 9a52924d-a727-4827-9f4c-da80e82c1a59 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95602025-04-30T06:38:35.811ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 9a52924d-a727-4827-9f4c-da80e82c1a59 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95612025-04-30T06:38:35.811ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 9a52924d-a727-4827-9f4c-da80e82c1a59 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95622025-04-30T06:38:35.812ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 9a52924d-a727-4827-9f4c-da80e82c1a59 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95632025-04-30T06:38:35.817ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc904a20-a558-4fb2-b9d1-9d999eba03b8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95642025-04-30T06:38:35.817ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
95652025-04-30T06:38:35.822ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62f7aa2c-1416-4a4f-bcd6-913e964b2103", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95662025-04-30T06:38:35.822ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
95672025-04-30T06:38:35.824ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bdb0fff-1ce2-4939-872a-616e2af9970f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95682025-04-30T06:38:35.824ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
95692025-04-30T06:38:35.825ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e518f4c-179a-42ac-86ae-76155f376b30", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95702025-04-30T06:38:35.825ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
95712025-04-30T06:38:35.826ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eea62166-eedf-4969-8faa-5b0285a047b7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95722025-04-30T06:38:35.826ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
95732025-04-30T06:38:35.827ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f647ab35-07d8-4924-a3f4-7d8750bcdc77", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95742025-04-30T06:38:35.827ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
95752025-04-30T06:38:35.828ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ba2f52-ac5d-4421-8f42-d833558f6033", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:34 GMT"} })
95762025-04-30T06:38:35.828ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
95772025-04-30T06:38:35.830ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18298f20-f92e-423b-b5a0-ec1931dd4bd4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
95782025-04-30T06:38:35.830ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
95792025-04-30T06:38:35.832ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ed0373f-ddfc-4418-a584-8f9bc9002e71", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
95802025-04-30T06:38:35.832ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
95812025-04-30T06:38:35.834ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6793cb17-6baf-49e0-9931-bf4fb4b129cd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
95822025-04-30T06:38:35.834ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
95832025-04-30T06:38:35.842ZINFOtest_quotas (dropshot_internal): request completed latency_us = 31473 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 9a52924d-a727-4827-9f4c-da80e82c1a59 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95842025-04-30T06:38:35.843ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a52924d-a727-4827-9f4c-da80e82c1a59", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
95852025-04-30T06:38:35.845ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
95862025-04-30T06:38:35.845ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95872025-04-30T06:38:35.845ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
95882025-04-30T06:38:35.857ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95892025-04-30T06:38:35.857ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95902025-04-30T06:38:35.858ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
95912025-04-30T06:38:35.861ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95922025-04-30T06:38:35.862ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95932025-04-30T06:38:35.862ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95942025-04-30T06:38:35.862ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95952025-04-30T06:38:35.868ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f21382fb-3dc5-4e40-a968-c2ac1a1c13ca", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
95962025-04-30T06:38:35.868ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
95972025-04-30T06:38:35.870ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23a54149-c545-44bb-82d8-360674e1f2a6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
95982025-04-30T06:38:35.870ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
95992025-04-30T06:38:35.872ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba3cc78b-e42f-4543-b50b-51aea505fb95", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96002025-04-30T06:38:35.873ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
96012025-04-30T06:38:35.931ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96022025-04-30T06:38:35.931ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96032025-04-30T06:38:35.931ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96042025-04-30T06:38:35.931ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
96052025-04-30T06:38:35.932ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96062025-04-30T06:38:35.936ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96072025-04-30T06:38:35.951ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96082025-04-30T06:38:35.951ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96092025-04-30T06:38:35.951ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
96102025-04-30T06:38:35.952ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96112025-04-30T06:38:35.994ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96122025-04-30T06:38:35.994ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
96132025-04-30T06:38:35.995ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96142025-04-30T06:38:36.023ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8764a88-1a2b-46b8-8305-f5fb8a911092", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96152025-04-30T06:38:36.023ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
96162025-04-30T06:38:36.033ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94ddd0cd-b69f-46b9-8737-319aa7155fed", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96172025-04-30T06:38:36.033ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
96182025-04-30T06:38:36.035ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e6f98bf-46db-4fa3-8801-90220ad7945f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96192025-04-30T06:38:36.036ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
96202025-04-30T06:38:36.039ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3e1fded-b229-4a87-a3cc-cdbc3966ff99", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96212025-04-30T06:38:36.039ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
96222025-04-30T06:38:36.040ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44bea55a-6f88-4146-b4d5-483d322cd4b6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96232025-04-30T06:38:36.040ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
96242025-04-30T06:38:36.046ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f9f9096-e8bb-412a-84b2-6e9f6889ebf2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96252025-04-30T06:38:36.054ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
96262025-04-30T06:38:36.085ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fccb7953-26d4-460b-9cdd-437051039d50", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96272025-04-30T06:38:36.086ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
96282025-04-30T06:38:36.089ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec2e819a-49d4-445e-af5c-c0ee8143535a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96292025-04-30T06:38:36.089ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
96302025-04-30T06:38:36.093ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc1c68e0-030c-496e-9d92-5e9def8841b7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96312025-04-30T06:38:36.093ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
96322025-04-30T06:38:36.095ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4e3371f-4653-4216-acba-ecb49ac75469", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96332025-04-30T06:38:36.096ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
96342025-04-30T06:38:36.098ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55c6c456-1451-47df-86a6-720098929cd7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96352025-04-30T06:38:36.102ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
96362025-04-30T06:38:36.103ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
96372025-04-30T06:38:36.103ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
96382025-04-30T06:38:36.104ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
96392025-04-30T06:38:36.115ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
96402025-04-30T06:38:36.124ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aaaa365-3462-4d45-b469-113b96f23f97", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96412025-04-30T06:38:36.125ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
96422025-04-30T06:38:36.125ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
96432025-04-30T06:38:36.125ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
96442025-04-30T06:38:36.127ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
96452025-04-30T06:38:36.141ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
96462025-04-30T06:38:36.141ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
96472025-04-30T06:38:36.141ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "114dee94-a77f-4cb7-88a6-e3fefdf837e3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96482025-04-30T06:38:36.141ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
96492025-04-30T06:38:36.143ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0114d2ed-caeb-43e8-b0b7-b7162afa8049", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96502025-04-30T06:38:36.143ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
96512025-04-30T06:38:36.145ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1f852eb-27e8-480c-b5e6-f447f116cd2f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96522025-04-30T06:38:36.145ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
96532025-04-30T06:38:36.147ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97ca0102-5476-4e79-9340-e9928e40f740", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96542025-04-30T06:38:36.147ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
96552025-04-30T06:38:36.168ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e481cd1b-0dcc-4855-ba45-4a518dfc0e75", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96562025-04-30T06:38:36.168ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
96572025-04-30T06:38:36.170ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d36a251-7fbb-471e-b57f-ae202acd2318", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96582025-04-30T06:38:36.170ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
96592025-04-30T06:38:36.172ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a97a21b-7ed7-4908-9ff7-89a6ecbe4b54", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96602025-04-30T06:38:36.172ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
96612025-04-30T06:38:36.174ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecc93c2c-e3b0-4faa-a356-31eb95663cdc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96622025-04-30T06:38:36.174ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
96632025-04-30T06:38:36.176ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0b8ce4d-9166-42c9-b02e-d7019863011a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96642025-04-30T06:38:36.176ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
96652025-04-30T06:38:36.178ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96662025-04-30T06:38:36.178ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96672025-04-30T06:38:36.178ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96682025-04-30T06:38:36.178ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
96692025-04-30T06:38:36.178ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96702025-04-30T06:38:36.179ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa144e22-0e7b-4a67-b381-65e0a371f2cb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96712025-04-30T06:38:36.179ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
96722025-04-30T06:38:36.181ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63cfed37-8ad1-459a-9928-ef8cbd643034", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96732025-04-30T06:38:36.181ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
96742025-04-30T06:38:36.182ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a90bbfa5-f85e-4b47-9831-7d6b5a5dfebc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96752025-04-30T06:38:36.182ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
96762025-04-30T06:38:36.183ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a6c39b6-5fb4-4d1f-a153-a1d36e4924fa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96772025-04-30T06:38:36.184ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
96782025-04-30T06:38:36.184ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41f52f20-e7c5-4771-9d12-6e321cfacade", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96792025-04-30T06:38:36.185ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
96802025-04-30T06:38:36.185ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d338bd7a-de6f-42bf-8101-58ac66e45074", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
96812025-04-30T06:38:36.185ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96822025-04-30T06:38:36.186ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96832025-04-30T06:38:36.186ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96842025-04-30T06:38:36.186ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96852025-04-30T06:38:36.186ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96862025-04-30T06:38:36.203ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96872025-04-30T06:38:36.222ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96882025-04-30T06:38:36.222ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96892025-04-30T06:38:36.223ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
96902025-04-30T06:38:36.223ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96912025-04-30T06:38:36.225ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
96922025-04-30T06:38:36.226ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96932025-04-30T06:38:36.245ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
96942025-04-30T06:38:36.245ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
96952025-04-30T06:38:36.245ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96962025-04-30T06:38:36.245ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96972025-04-30T06:38:36.245ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96982025-04-30T06:38:36.246ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96992025-04-30T06:38:36.252ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97002025-04-30T06:38:36.252ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
97012025-04-30T06:38:36.253ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97022025-04-30T06:38:36.262ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97032025-04-30T06:38:36.274ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
97042025-04-30T06:38:36.274ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97052025-04-30T06:38:36.274ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97062025-04-30T06:38:36.276ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97072025-04-30T06:38:36.490ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
97082025-04-30T06:38:36.490ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
97092025-04-30T06:38:36.491ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:53064 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
97102025-04-30T06:38:36.491ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:53064 req_id = 56dcdc92-66cf-478c-a314-73a8b5fa346c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97112025-04-30T06:38:36.492ZINFOtest_quotas (dropshot): request completed latency_us = 543 local_addr = [::1]:55655 method = GET remote_addr = [::1]:53064 req_id = 56dcdc92-66cf-478c-a314-73a8b5fa346c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97122025-04-30T06:38:36.492ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "56dcdc92-66cf-478c-a314-73a8b5fa346c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97132025-04-30T06:38:36.494ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97142025-04-30T06:38:36.495ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97152025-04-30T06:38:36.495ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
97162025-04-30T06:38:36.496ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97172025-04-30T06:38:36.496ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97182025-04-30T06:38:36.496ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97192025-04-30T06:38:36.496ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97202025-04-30T06:38:36.497ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97212025-04-30T06:38:36.516ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97222025-04-30T06:38:36.520ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97232025-04-30T06:38:36.520ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97242025-04-30T06:38:36.520ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97252025-04-30T06:38:36.520ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97262025-04-30T06:38:36.632ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
97272025-04-30T06:38:36.636ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
97282025-04-30T06:38:36.637ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
97292025-04-30T06:38:36.637ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.108281128s iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
97302025-04-30T06:38:36.637ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
97312025-04-30T06:38:36.637ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97322025-04-30T06:38:36.637ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97332025-04-30T06:38:36.637ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97342025-04-30T06:38:36.637ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97352025-04-30T06:38:36.638ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97362025-04-30T06:38:36.646ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97372025-04-30T06:38:36.650ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97382025-04-30T06:38:36.650ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97392025-04-30T06:38:36.650ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97402025-04-30T06:38:36.651ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97412025-04-30T06:38:36.667ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
97422025-04-30T06:38:36.667ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
97432025-04-30T06:38:36.669ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 24 4f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 $O..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61349 req_id = 20285678-08da-40e9-b5dc-97647f12a4f8
97442025-04-30T06:38:36.669ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9295,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61349 req_id = 20285678-08da-40e9-b5dc-97647f12a4f8
97452025-04-30T06:38:36.669ZDEBGtest_quotas (store): query key key = _mgs._tcp
97462025-04-30T06:38:36.669ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97472025-04-30T06:38:36.669ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61349 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 20285678-08da-40e9-b5dc-97647f12a4f8
97482025-04-30T06:38:36.669ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3981, tv_nsec: 468838837 } })
97492025-04-30T06:38:36.670ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
97502025-04-30T06:38:36.670ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d2 d3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59642 req_id = ece7e5f2-ff0a-4de2-b7d0-ec41fa895ba0
97512025-04-30T06:38:36.670ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53971,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59642 req_id = ece7e5f2-ff0a-4de2-b7d0-ec41fa895ba0
97522025-04-30T06:38:36.670ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97532025-04-30T06:38:36.670ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59642 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ece7e5f2-ff0a-4de2-b7d0-ec41fa895ba0
97542025-04-30T06:38:36.719ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
97552025-04-30T06:38:36.720ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97562025-04-30T06:38:36.720ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97572025-04-30T06:38:36.720ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97582025-04-30T06:38:36.720ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
97592025-04-30T06:38:36.720ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97602025-04-30T06:38:36.721ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9f fc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51940 req_id = 15da20c1-6feb-4fa5-ab94-706a33597238
97612025-04-30T06:38:36.721ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40956,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51940 req_id = 15da20c1-6feb-4fa5-ab94-706a33597238
97622025-04-30T06:38:36.721ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
97632025-04-30T06:38:36.721ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40956, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51940 req_id = 15da20c1-6feb-4fa5-ab94-706a33597238
97642025-04-30T06:38:36.721ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
97652025-04-30T06:38:36.721ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
97662025-04-30T06:38:36.721ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
97672025-04-30T06:38:36.732ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bd75c9a-8ee8-4c39-8d47-38d463894b2d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97682025-04-30T06:38:36.732ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
97692025-04-30T06:38:36.732ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97702025-04-30T06:38:36.738ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97712025-04-30T06:38:36.738ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97722025-04-30T06:38:36.738ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
97732025-04-30T06:38:36.738ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97742025-04-30T06:38:36.742ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97752025-04-30T06:38:36.743ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
97762025-04-30T06:38:36.743ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97772025-04-30T06:38:36.745ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12e6b4bf-9d0f-4264-ba56-72bb866acc50", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97782025-04-30T06:38:36.746ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
97792025-04-30T06:38:36.749ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17cc8d49-69bf-472d-bc14-c28d3459475c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97802025-04-30T06:38:36.750ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
97812025-04-30T06:38:36.752ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43d9e58b-3bf8-4cb3-abea-89a62b8564a5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97822025-04-30T06:38:36.752ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
97832025-04-30T06:38:36.755ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db10b827-820f-4bd0-8cda-f591c8650867", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97842025-04-30T06:38:36.755ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
97852025-04-30T06:38:36.757ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ed57ee0-849e-47e4-b6e7-4f1c4d4ff946", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97862025-04-30T06:38:36.758ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
97872025-04-30T06:38:36.760ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e9c770d-df8a-4f8d-a68c-8d404980c4c3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97882025-04-30T06:38:36.760ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
97892025-04-30T06:38:36.765ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fed97d6-0a13-4230-a893-03de39021f13", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97902025-04-30T06:38:36.765ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
97912025-04-30T06:38:36.767ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "912f958f-3c5f-43af-978e-02202abdf7a0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97922025-04-30T06:38:36.767ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
97932025-04-30T06:38:36.768ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cad3458-8fba-455e-8b3d-3a19e04045b8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97942025-04-30T06:38:36.770ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
97952025-04-30T06:38:36.771ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0bfc4ec-c77b-4383-ac9b-51be42be8c38", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97962025-04-30T06:38:36.771ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
97972025-04-30T06:38:36.774ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d200854f-fc87-4071-a001-c73c5dd60fdb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
97982025-04-30T06:38:36.774ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
97992025-04-30T06:38:36.775ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
98002025-04-30T06:38:36.776ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce57f321-2e64-4941-bcfa-f76dece9a7d4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
98012025-04-30T06:38:36.777ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
98022025-04-30T06:38:36.779ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e77e639f-b927-44b3-b146-c847bde58bb7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
98032025-04-30T06:38:36.779ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
98042025-04-30T06:38:36.781ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8f40641-202f-483c-87ec-ae6fc78f9f4c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
98052025-04-30T06:38:36.782ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
98062025-04-30T06:38:36.792ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f80854a-4e33-4626-9e42-554f3e48d22e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
98072025-04-30T06:38:36.792ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
98082025-04-30T06:38:36.796ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62cfeeff-d190-442f-ab27-6aca33d2836f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
98092025-04-30T06:38:36.796ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
98102025-04-30T06:38:36.799ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11a1fbe8-ff6a-4914-a4c0-815be66cf0a5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
98112025-04-30T06:38:36.799ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
98122025-04-30T06:38:36.801ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "26c982a5-7f32-4b1b-a398-218725941bc1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
98132025-04-30T06:38:36.801ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
98142025-04-30T06:38:36.802ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a21d80f2-c4fc-4754-9a85-725e210bd06b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
98152025-04-30T06:38:36.803ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
98162025-04-30T06:38:36.816ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b7d9e2a-cf51-459d-ab0a-7ca2c2efd448", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
98172025-04-30T06:38:36.816ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
98182025-04-30T06:38:36.825ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5c17725-7bdb-45e5-8a96-344dd1ca259c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:35 GMT"} })
98192025-04-30T06:38:36.825ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
98202025-04-30T06:38:36.826ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98212025-04-30T06:38:36.826ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98222025-04-30T06:38:36.826ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98232025-04-30T06:38:36.826ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
98242025-04-30T06:38:36.827ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98252025-04-30T06:38:36.832ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5908df4-90cb-4b7a-9d0b-d66280263426", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98262025-04-30T06:38:36.833ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
98272025-04-30T06:38:36.840ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7878d0b9-d1d2-4b66-b9bc-9b4fa7d58927", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98282025-04-30T06:38:36.840ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
98292025-04-30T06:38:36.841ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98302025-04-30T06:38:36.848ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98312025-04-30T06:38:36.848ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98322025-04-30T06:38:36.848ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
98332025-04-30T06:38:36.849ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98342025-04-30T06:38:36.850ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
98352025-04-30T06:38:36.850ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
98362025-04-30T06:38:36.902ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98372025-04-30T06:38:36.902ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
98382025-04-30T06:38:36.903ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98392025-04-30T06:38:36.904ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
98402025-04-30T06:38:36.904ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
98412025-04-30T06:38:36.905ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
98422025-04-30T06:38:36.905ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
98432025-04-30T06:38:36.906ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c5eda4c-7cc5-4886-a9ed-0cf9f572f47b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98442025-04-30T06:38:36.907ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
98452025-04-30T06:38:36.925ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f5ddbe7-dd11-4c81-8c3e-6ab2e36f7f27", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98462025-04-30T06:38:36.925ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
98472025-04-30T06:38:36.952ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfe4b208-d161-42f1-ae73-e9a749cd4d36", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98482025-04-30T06:38:36.952ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
98492025-04-30T06:38:36.965ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "542041d7-5e65-45b7-852b-123b3157b7d0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98502025-04-30T06:38:36.965ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
98512025-04-30T06:38:36.966ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
98522025-04-30T06:38:36.968ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85397ed6-76cf-4bed-bfb7-77f257f865f1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98532025-04-30T06:38:36.968ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
98542025-04-30T06:38:36.971ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee2a5937-071f-4608-8184-8f43782c6358", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98552025-04-30T06:38:36.971ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
98562025-04-30T06:38:36.972ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e98d980-2253-43d9-bc9e-18938b00b82e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98572025-04-30T06:38:36.972ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
98582025-04-30T06:38:36.974ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1a71aa3-cd35-46a5-a4d2-84ba57932a0f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98592025-04-30T06:38:36.974ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
98602025-04-30T06:38:36.976ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65fb87f0-61ac-47a1-bf93-11bb87c89d58", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98612025-04-30T06:38:36.976ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
98622025-04-30T06:38:36.983ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "016d1313-508c-48e2-9f70-23de5c83a4b2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98632025-04-30T06:38:36.983ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
98642025-04-30T06:38:36.985ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7cd3d54-89e3-4b20-a6de-4f255409eba3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98652025-04-30T06:38:36.985ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
98662025-04-30T06:38:36.988ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d8724b5-52a9-4059-b6dc-f06bf7191df3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98672025-04-30T06:38:36.988ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
98682025-04-30T06:38:36.990ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e111ba5-7244-4cfa-b039-733a55171d18", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98692025-04-30T06:38:36.990ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
98702025-04-30T06:38:36.992ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba43cf68-bc19-439d-ac67-149f75c467a9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98712025-04-30T06:38:36.992ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
98722025-04-30T06:38:36.995ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa118331-b2df-4b96-8946-551c8ccfbb0b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98732025-04-30T06:38:36.995ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
98742025-04-30T06:38:37.004ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dd00f12-16e4-4cc7-a395-4be9665bfe5c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98752025-04-30T06:38:37.004ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
98762025-04-30T06:38:37.004ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb07c5ee-4613-40bc-8ce3-1af4d4fafe53", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98772025-04-30T06:38:37.004ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
98782025-04-30T06:38:37.011ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dee4c498-d044-43ac-9cbe-2e1d0349fdf6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98792025-04-30T06:38:37.011ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
98802025-04-30T06:38:37.013ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f8ae848-c753-476b-af20-759dbd3f1ff4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98812025-04-30T06:38:37.013ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
98822025-04-30T06:38:37.014ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cf01811-4754-4e8b-825f-25866637bfe3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98832025-04-30T06:38:37.015ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
98842025-04-30T06:38:37.017ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98852025-04-30T06:38:37.017ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98862025-04-30T06:38:37.017ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98872025-04-30T06:38:37.017ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
98882025-04-30T06:38:37.018ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98892025-04-30T06:38:37.020ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a30680d-5c3c-4938-bf31-d26cd6aa2d77", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
98902025-04-30T06:38:37.020ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
98912025-04-30T06:38:37.020ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98922025-04-30T06:38:37.020ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98932025-04-30T06:38:37.020ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98942025-04-30T06:38:37.021ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98952025-04-30T06:38:37.022ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98962025-04-30T06:38:37.030ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98972025-04-30T06:38:37.030ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98982025-04-30T06:38:37.030ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
98992025-04-30T06:38:37.030ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99002025-04-30T06:38:37.033ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99012025-04-30T06:38:37.040ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
99022025-04-30T06:38:37.040ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
99032025-04-30T06:38:37.040ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99042025-04-30T06:38:37.040ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99052025-04-30T06:38:37.040ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99062025-04-30T06:38:37.040ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99072025-04-30T06:38:37.061ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99082025-04-30T06:38:37.061ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
99092025-04-30T06:38:37.062ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99102025-04-30T06:38:37.064ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99112025-04-30T06:38:37.069ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
99122025-04-30T06:38:37.069ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99132025-04-30T06:38:37.069ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99142025-04-30T06:38:37.070ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99152025-04-30T06:38:37.174ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
99162025-04-30T06:38:37.175ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
99172025-04-30T06:38:37.176ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:45726 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
99182025-04-30T06:38:37.176ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99192025-04-30T06:38:37.177ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99202025-04-30T06:38:37.177ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
99212025-04-30T06:38:37.178ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
99222025-04-30T06:38:37.182ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:45726 req_id = 33e4808e-8bc3-46f6-b582-0b2ce365c119 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99232025-04-30T06:38:37.183ZINFOtest_quotas (dropshot): request completed latency_us = 582 local_addr = [::1]:55655 method = GET remote_addr = [::1]:45726 req_id = 33e4808e-8bc3-46f6-b582-0b2ce365c119 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99242025-04-30T06:38:37.183ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
99252025-04-30T06:38:37.183ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
99262025-04-30T06:38:37.184ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "33e4808e-8bc3-46f6-b582-0b2ce365c119", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
99272025-04-30T06:38:37.186ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99282025-04-30T06:38:37.187ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99292025-04-30T06:38:37.187ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
99302025-04-30T06:38:37.188ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99312025-04-30T06:38:37.189ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99322025-04-30T06:38:37.189ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99332025-04-30T06:38:37.189ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99342025-04-30T06:38:37.189ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99352025-04-30T06:38:37.193ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
99362025-04-30T06:38:37.193ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99372025-04-30T06:38:37.195ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99382025-04-30T06:38:37.200ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99392025-04-30T06:38:37.200ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99402025-04-30T06:38:37.200ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99412025-04-30T06:38:37.201ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99422025-04-30T06:38:37.225ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
99432025-04-30T06:38:37.237ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
99442025-04-30T06:38:37.237ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
99452025-04-30T06:38:37.237ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 600.563644ms iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
99462025-04-30T06:38:37.237ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
99472025-04-30T06:38:37.238ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99482025-04-30T06:38:37.238ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99492025-04-30T06:38:37.238ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99502025-04-30T06:38:37.238ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99512025-04-30T06:38:37.239ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99522025-04-30T06:38:37.241ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99532025-04-30T06:38:37.246ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99542025-04-30T06:38:37.246ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99552025-04-30T06:38:37.246ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99562025-04-30T06:38:37.246ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99572025-04-30T06:38:37.249ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
99582025-04-30T06:38:37.249ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
99592025-04-30T06:38:37.250ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: dc b8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59891 req_id = 32b0195b-6853-4f5e-9e69-18b147562862
99602025-04-30T06:38:37.250ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56504,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59891 req_id = 32b0195b-6853-4f5e-9e69-18b147562862
99612025-04-30T06:38:37.250ZDEBGtest_quotas (store): query key key = _mgs._tcp
99622025-04-30T06:38:37.251ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99632025-04-30T06:38:37.251ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59891 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 32b0195b-6853-4f5e-9e69-18b147562862
99642025-04-30T06:38:37.251ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3982, tv_nsec: 50397625 } })
99652025-04-30T06:38:37.251ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
99662025-04-30T06:38:37.251ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9a 5c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .\\..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60773 req_id = 8d57d1a8-d487-474b-890d-f8e6ae34bfff
99672025-04-30T06:38:37.251ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39516,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60773 req_id = 8d57d1a8-d487-474b-890d-f8e6ae34bfff
99682025-04-30T06:38:37.252ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99692025-04-30T06:38:37.252ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60773 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8d57d1a8-d487-474b-890d-f8e6ae34bfff
99702025-04-30T06:38:37.306ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
99712025-04-30T06:38:37.307ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99722025-04-30T06:38:37.307ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99732025-04-30T06:38:37.307ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99742025-04-30T06:38:37.307ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
99752025-04-30T06:38:37.307ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99762025-04-30T06:38:37.308ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
99772025-04-30T06:38:37.353ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 17 5c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .\\..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59519 req_id = eab3f0f9-42d5-46ed-8c71-f8a98cf79df8
99782025-04-30T06:38:37.353ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5980,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59519 req_id = eab3f0f9-42d5-46ed-8c71-f8a98cf79df8
99792025-04-30T06:38:37.364ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
99802025-04-30T06:38:37.364ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5980, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59519 req_id = eab3f0f9-42d5-46ed-8c71-f8a98cf79df8
99812025-04-30T06:38:37.367ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
99822025-04-30T06:38:37.367ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
99832025-04-30T06:38:37.367ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
99842025-04-30T06:38:37.368ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99852025-04-30T06:38:37.376ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99862025-04-30T06:38:37.377ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99872025-04-30T06:38:37.377ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
99882025-04-30T06:38:37.377ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99892025-04-30T06:38:37.379ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "70ba685f-21b9-4f1a-bf03-4a54b8b34828", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
99902025-04-30T06:38:37.379ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
99912025-04-30T06:38:37.385ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99922025-04-30T06:38:37.385ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
99932025-04-30T06:38:37.394ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99942025-04-30T06:38:37.395ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d557dae-fadb-44c3-8b00-f02557feaf16", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
99952025-04-30T06:38:37.395ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
99962025-04-30T06:38:37.399ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "078622a5-1f33-4996-9203-3acc6201cfd8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
99972025-04-30T06:38:37.399ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
99982025-04-30T06:38:37.408ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "427fd632-6122-4830-9863-fbeced20afa5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
99992025-04-30T06:38:37.408ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
100002025-04-30T06:38:37.411ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d875d5d7-0b9b-4e0e-9929-415d082311d9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100012025-04-30T06:38:37.411ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
100022025-04-30T06:38:37.413ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39f4ead8-6c67-4623-9bb6-7b4f0b0d5109", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100032025-04-30T06:38:37.413ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
100042025-04-30T06:38:37.416ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eac33b74-74ea-4088-a5f2-4e1c097d91c4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100052025-04-30T06:38:37.416ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
100062025-04-30T06:38:37.419ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b709ff3-592a-4330-80c3-d08cf22703be", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100072025-04-30T06:38:37.420ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
100082025-04-30T06:38:37.422ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "776b7b29-cc4e-4174-ab7b-e4d3b10562dc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100092025-04-30T06:38:37.422ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
100102025-04-30T06:38:37.423ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e41b2519-3d08-490b-b7c2-88a56e6d6230", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100112025-04-30T06:38:37.423ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
100122025-04-30T06:38:37.427ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc6b1c76-9b10-4361-83cd-b8b94ecdbdee", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100132025-04-30T06:38:37.427ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
100142025-04-30T06:38:37.428ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
100152025-04-30T06:38:37.429ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bdf1ae7-049b-43b0-a60d-3d5968e3eaa8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100162025-04-30T06:38:37.429ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
100172025-04-30T06:38:37.430ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9518322e-19c9-4309-b51f-34cc8c1798e9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100182025-04-30T06:38:37.430ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
100192025-04-30T06:38:37.437ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e3f53f2-ecbc-4082-844e-cbca9944931f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100202025-04-30T06:38:37.437ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
100212025-04-30T06:38:37.440ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e615230-b299-496e-9a3a-6615c7b0ef03", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100222025-04-30T06:38:37.440ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
100232025-04-30T06:38:37.453ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8ed8b2c-f4b2-4237-a8c1-8c45bff17909", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100242025-04-30T06:38:37.453ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
100252025-04-30T06:38:37.459ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2f26fb5-e9e7-4100-8a72-198889b06b8b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100262025-04-30T06:38:37.460ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
100272025-04-30T06:38:37.462ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2a3f679-7347-4f8c-a834-ae1c60d22e5d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100282025-04-30T06:38:37.463ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
100292025-04-30T06:38:37.465ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6b0167a-a5e2-4e33-ab30-d9037b1b4d03", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100302025-04-30T06:38:37.466ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
100312025-04-30T06:38:37.466ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a5bc6aa-7271-4bb4-b0b5-a01057c5d1a9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100322025-04-30T06:38:37.467ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
100332025-04-30T06:38:37.468ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9328d0d-a7cb-4ece-b99b-5e7f6a3e7f7c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100342025-04-30T06:38:37.468ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
100352025-04-30T06:38:37.472ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dd413fe-981b-48b1-84d7-931cb5b6a0da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100362025-04-30T06:38:37.473ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
100372025-04-30T06:38:37.474ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2604d4e-84f0-4e6b-b702-0dad967bc68e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100382025-04-30T06:38:37.474ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
100392025-04-30T06:38:37.475ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "722eb4be-330e-4704-908d-d7cf70636f6a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100402025-04-30T06:38:37.475ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
100412025-04-30T06:38:37.479ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100422025-04-30T06:38:37.479ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100432025-04-30T06:38:37.479ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100442025-04-30T06:38:37.479ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
100452025-04-30T06:38:37.481ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100462025-04-30T06:38:37.482ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2f2d570-42c3-4cf0-bd47-5c8c8e20cdf0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100472025-04-30T06:38:37.482ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
100482025-04-30T06:38:37.484ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c186b4ca-5b8c-413d-bd63-6e47c2c8ef71", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100492025-04-30T06:38:37.484ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
100502025-04-30T06:38:37.485ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100512025-04-30T06:38:37.491ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100522025-04-30T06:38:37.491ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100532025-04-30T06:38:37.491ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
100542025-04-30T06:38:37.492ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100552025-04-30T06:38:37.495ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4251e407-d9f7-48c4-bd29-fcf626fa4f3a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100562025-04-30T06:38:37.495ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
100572025-04-30T06:38:37.495ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100582025-04-30T06:38:37.495ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
100592025-04-30T06:38:37.496ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100602025-04-30T06:38:37.498ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5728f83a-637a-4cde-ae00-8e52aee21a76", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100612025-04-30T06:38:37.498ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
100622025-04-30T06:38:37.501ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb9b5d58-4c59-4d1e-bfac-f9f6694e5208", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100632025-04-30T06:38:37.501ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
100642025-04-30T06:38:37.504ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "616b6570-a00a-4054-8f5e-364924b49cd3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100652025-04-30T06:38:37.504ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
100662025-04-30T06:38:37.505ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaf4400a-8363-4901-8248-3da20360a508", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100672025-04-30T06:38:37.505ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
100682025-04-30T06:38:37.508ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "306ecb27-5410-4a40-ba60-0e741c44f608", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100692025-04-30T06:38:37.508ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
100702025-04-30T06:38:37.510ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5b61c21-6b3a-461c-8316-0fb64c9cbe32", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100712025-04-30T06:38:37.510ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
100722025-04-30T06:38:37.514ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c610171-62c1-45b9-af98-b7812ab52047", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100732025-04-30T06:38:37.514ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
100742025-04-30T06:38:37.516ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a5f6d6f-8167-46c5-bdce-c8734ecb129a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100752025-04-30T06:38:37.517ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
100762025-04-30T06:38:37.519ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52bbdb7d-486b-4882-9fc8-33e14974ae67", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100772025-04-30T06:38:37.520ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
100782025-04-30T06:38:37.523ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fda99e04-f512-481a-94b7-56841a59c132", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100792025-04-30T06:38:37.524ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
100802025-04-30T06:38:37.540ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
100812025-04-30T06:38:37.555ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f988bfe-f173-4a27-82ae-022d8cdc0a8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100822025-04-30T06:38:37.556ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
100832025-04-30T06:38:37.558ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdca45c0-a407-449d-a634-eae26796aaad", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100842025-04-30T06:38:37.566ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
100852025-04-30T06:38:37.568ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7ba1028-9f03-4536-a620-cac6276fbb92", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100862025-04-30T06:38:37.593ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
100872025-04-30T06:38:37.610ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100882025-04-30T06:38:37.610ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100892025-04-30T06:38:37.610ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100902025-04-30T06:38:37.610ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
100912025-04-30T06:38:37.611ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100922025-04-30T06:38:37.615ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100932025-04-30T06:38:37.627ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100942025-04-30T06:38:37.627ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100952025-04-30T06:38:37.627ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
100962025-04-30T06:38:37.627ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100972025-04-30T06:38:37.628ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3f448ed-b03d-4333-bb27-5f2163f1a3fd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
100982025-04-30T06:38:37.628ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
100992025-04-30T06:38:37.630ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35609ba7-ed9a-4191-b227-8e812317c8a9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
101002025-04-30T06:38:37.630ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
101012025-04-30T06:38:37.631ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101022025-04-30T06:38:37.631ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
101032025-04-30T06:38:37.631ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101042025-04-30T06:38:37.632ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cc07a9f-13c0-42df-8cd3-88731cd82c46", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
101052025-04-30T06:38:37.632ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
101062025-04-30T06:38:37.633ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74c9f0f6-512c-4385-97f2-48502a310db3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
101072025-04-30T06:38:37.634ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
101082025-04-30T06:38:37.635ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4604306-3cd6-4b6c-81d1-463a425b755d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
101092025-04-30T06:38:37.635ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101102025-04-30T06:38:37.635ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101112025-04-30T06:38:37.635ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101122025-04-30T06:38:37.635ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101132025-04-30T06:38:37.635ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101142025-04-30T06:38:37.640ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101152025-04-30T06:38:37.645ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101162025-04-30T06:38:37.645ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101172025-04-30T06:38:37.645ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101182025-04-30T06:38:37.645ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101192025-04-30T06:38:37.645ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101202025-04-30T06:38:37.646ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101212025-04-30T06:38:37.649ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101222025-04-30T06:38:37.654ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101232025-04-30T06:38:37.655ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101242025-04-30T06:38:37.655ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101252025-04-30T06:38:37.656ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101262025-04-30T06:38:37.708ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
101272025-04-30T06:38:37.708ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
101282025-04-30T06:38:37.709ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:63153 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
101292025-04-30T06:38:37.710ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:63153 req_id = fbae053e-ed1c-486c-90b5-e1a551daac84 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101302025-04-30T06:38:37.711ZINFOtest_quotas (dropshot): request completed latency_us = 663 local_addr = [::1]:55655 method = GET remote_addr = [::1]:63153 req_id = fbae053e-ed1c-486c-90b5-e1a551daac84 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101312025-04-30T06:38:37.711ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbae053e-ed1c-486c-90b5-e1a551daac84", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
101322025-04-30T06:38:37.714ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101332025-04-30T06:38:37.714ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101342025-04-30T06:38:37.714ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
101352025-04-30T06:38:37.716ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101362025-04-30T06:38:37.716ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101372025-04-30T06:38:37.716ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101382025-04-30T06:38:37.717ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101392025-04-30T06:38:37.717ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101402025-04-30T06:38:37.720ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101412025-04-30T06:38:37.724ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101422025-04-30T06:38:37.724ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101432025-04-30T06:38:37.724ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101442025-04-30T06:38:37.725ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101452025-04-30T06:38:37.739ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
101462025-04-30T06:38:37.740ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101472025-04-30T06:38:37.740ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 116ca842-d1d9-4df6-9b5d-888d05426c94 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101482025-04-30T06:38:37.741ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 116ca842-d1d9-4df6-9b5d-888d05426c94 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101492025-04-30T06:38:37.741ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 116ca842-d1d9-4df6-9b5d-888d05426c94 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101502025-04-30T06:38:37.741ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 116ca842-d1d9-4df6-9b5d-888d05426c94 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101512025-04-30T06:38:37.742ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
101522025-04-30T06:38:37.745ZINFOtest_quotas (dropshot_internal): request completed latency_us = 4469 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 116ca842-d1d9-4df6-9b5d-888d05426c94 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
101532025-04-30T06:38:37.745ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "116ca842-d1d9-4df6-9b5d-888d05426c94", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
101542025-04-30T06:38:37.745ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
101552025-04-30T06:38:37.748ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = bd43692d-9a31-448c-b6b2-b772dedd9b7f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
101562025-04-30T06:38:37.749ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = bd43692d-9a31-448c-b6b2-b772dedd9b7f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
101572025-04-30T06:38:37.749ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = bd43692d-9a31-448c-b6b2-b772dedd9b7f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
101582025-04-30T06:38:37.749ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = bd43692d-9a31-448c-b6b2-b772dedd9b7f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
101592025-04-30T06:38:37.755ZINFOtest_quotas (dropshot_internal): request completed latency_us = 6264 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = bd43692d-9a31-448c-b6b2-b772dedd9b7f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
101602025-04-30T06:38:37.755ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd43692d-9a31-448c-b6b2-b772dedd9b7f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:36 GMT"} })
101612025-04-30T06:38:37.755ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
101622025-04-30T06:38:37.755ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101632025-04-30T06:38:37.755ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
101642025-04-30T06:38:37.763ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
101652025-04-30T06:38:37.764ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
101662025-04-30T06:38:37.764ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 526.253035ms iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
101672025-04-30T06:38:37.764ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
101682025-04-30T06:38:37.764ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101692025-04-30T06:38:37.764ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101702025-04-30T06:38:37.764ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101712025-04-30T06:38:37.764ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101722025-04-30T06:38:37.765ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101732025-04-30T06:38:37.767ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101742025-04-30T06:38:37.782ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101752025-04-30T06:38:37.782ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101762025-04-30T06:38:37.783ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101772025-04-30T06:38:37.784ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101782025-04-30T06:38:37.787ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
101792025-04-30T06:38:37.787ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
101802025-04-30T06:38:37.788ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a3 13 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52348 req_id = 636c4612-0006-4e7a-b45d-e74ff353047f
101812025-04-30T06:38:37.788ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41747,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52348 req_id = 636c4612-0006-4e7a-b45d-e74ff353047f
101822025-04-30T06:38:37.788ZDEBGtest_quotas (store): query key key = _mgs._tcp
101832025-04-30T06:38:37.789ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101842025-04-30T06:38:37.789ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52348 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 636c4612-0006-4e7a-b45d-e74ff353047f
101852025-04-30T06:38:37.790ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3982, tv_nsec: 588940165 } })
101862025-04-30T06:38:37.790ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
101872025-04-30T06:38:37.790ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 66 d0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e f...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63889 req_id = 44101241-84ed-4e22-8054-bb0fda9e0621
101882025-04-30T06:38:37.790ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26320,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63889 req_id = 44101241-84ed-4e22-8054-bb0fda9e0621
101892025-04-30T06:38:37.790ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101902025-04-30T06:38:37.792ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63889 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 44101241-84ed-4e22-8054-bb0fda9e0621
101912025-04-30T06:38:37.793ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101922025-04-30T06:38:37.793ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101932025-04-30T06:38:37.793ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101942025-04-30T06:38:37.793ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
101952025-04-30T06:38:37.793ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101962025-04-30T06:38:37.851ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
101972025-04-30T06:38:37.852ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101982025-04-30T06:38:37.852ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
101992025-04-30T06:38:37.853ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f2 b7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55149 req_id = d4a46277-3ca7-4132-94a1-8a326211feb9
102002025-04-30T06:38:37.853ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62135,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55149 req_id = d4a46277-3ca7-4132-94a1-8a326211feb9
102012025-04-30T06:38:37.853ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
102022025-04-30T06:38:37.853ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62135, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55149 req_id = d4a46277-3ca7-4132-94a1-8a326211feb9
102032025-04-30T06:38:37.854ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102042025-04-30T06:38:37.863ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102052025-04-30T06:38:37.863ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102062025-04-30T06:38:37.863ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
102072025-04-30T06:38:37.863ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102082025-04-30T06:38:37.863ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
102092025-04-30T06:38:37.863ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
102102025-04-30T06:38:37.863ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
102112025-04-30T06:38:37.864ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102122025-04-30T06:38:37.864ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102132025-04-30T06:38:37.864ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102142025-04-30T06:38:37.864ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102152025-04-30T06:38:37.867ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102162025-04-30T06:38:37.867ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
102172025-04-30T06:38:37.867ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102182025-04-30T06:38:37.868ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b54a0445-7990-4523-808a-48d2077ccbb3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102192025-04-30T06:38:37.869ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
102202025-04-30T06:38:37.870ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c43cd4bb-1f8f-43a9-9a80-3c0ad5d3b2a0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102212025-04-30T06:38:37.870ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
102222025-04-30T06:38:37.874ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d5c1f1d-3a9d-4406-9246-1771134390e1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102232025-04-30T06:38:37.883ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
102242025-04-30T06:38:37.883ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a95e14fd-4c44-4ec7-9bd6-c895fc698262", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102252025-04-30T06:38:37.883ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
102262025-04-30T06:38:37.891ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "976a482f-fe97-4f82-b270-feb78a8b0e2d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102272025-04-30T06:38:37.891ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
102282025-04-30T06:38:37.891ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59d65495-2a04-473b-8a44-06c6c69fed95", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102292025-04-30T06:38:37.891ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
102302025-04-30T06:38:37.891ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ebec25d-179e-4dc1-9065-f469670e80cf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102312025-04-30T06:38:37.891ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
102322025-04-30T06:38:37.891ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9479cf5c-b4de-4e20-98ce-b489e93d09bc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102332025-04-30T06:38:37.891ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
102342025-04-30T06:38:37.895ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ead8d7f-13ec-4c45-8c5a-fe3596903a4d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102352025-04-30T06:38:37.895ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
102362025-04-30T06:38:37.899ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee3f0759-f44b-4022-a771-bac4490b0ff1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102372025-04-30T06:38:37.899ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
102382025-04-30T06:38:37.900ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce6610a6-88e2-4455-884e-e2abf1269c4b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102392025-04-30T06:38:37.901ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
102402025-04-30T06:38:37.902ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf140d7f-b2d5-479d-91f1-82fa3055b41d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102412025-04-30T06:38:37.902ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
102422025-04-30T06:38:37.904ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cce51a8-c19e-4c32-929b-81dbb462535a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102432025-04-30T06:38:37.904ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
102442025-04-30T06:38:37.910ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afedab45-69c4-4b41-91dc-62679eeaa7f3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102452025-04-30T06:38:37.911ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
102462025-04-30T06:38:37.914ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48511d51-82c3-4341-82bd-f0102b2e6d53", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102472025-04-30T06:38:37.914ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
102482025-04-30T06:38:37.918ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46f31f2e-44c3-4c07-907a-7f8ab1decaf3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102492025-04-30T06:38:37.918ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
102502025-04-30T06:38:37.924ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "927b2b8c-5e7c-4575-87ce-3a23a26080d2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
102512025-04-30T06:38:37.924ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
102522025-04-30T06:38:37.947ZDEBGtest_quotas (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
102532025-04-30T06:38:37.947ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
102542025-04-30T06:38:37.948ZDEBGtest_quotas (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
102552025-04-30T06:38:37.948ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
102562025-04-30T06:38:37.949ZDEBGtest_quotas (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
102572025-04-30T06:38:37.949ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
102582025-04-30T06:38:37.949ZDEBGtest_quotas (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
102592025-04-30T06:38:37.949ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
102602025-04-30T06:38:37.950ZDEBGtest_quotas (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
102612025-04-30T06:38:37.950ZDEBGtest_quotas (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
102622025-04-30T06:38:37.950ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 103.466\u{b5}s iteration = 3 status = {"error":"task disabled"}
102632025-04-30T06:38:37.950ZDEBGtest_quotas (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
102642025-04-30T06:38:37.950ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
102652025-04-30T06:38:37.950ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102662025-04-30T06:38:37.950ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
102672025-04-30T06:38:37.950ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102682025-04-30T06:38:37.951ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102692025-04-30T06:38:37.951ZDEBGtest_quotas (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
102702025-04-30T06:38:37.951ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
102712025-04-30T06:38:37.951ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102722025-04-30T06:38:37.951ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
102732025-04-30T06:38:37.951ZDEBGtest_quotas (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
102742025-04-30T06:38:37.952ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
102752025-04-30T06:38:37.952ZDEBGtest_quotas (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
102762025-04-30T06:38:37.952ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
102772025-04-30T06:38:37.952ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
102782025-04-30T06:38:37.953ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
102792025-04-30T06:38:37.953ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
102802025-04-30T06:38:37.954ZDEBGtest_quotas (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
102812025-04-30T06:38:37.954ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
102822025-04-30T06:38:37.954ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102832025-04-30T06:38:37.954ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
102842025-04-30T06:38:37.955ZDEBGtest_quotas (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
102852025-04-30T06:38:37.955ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
102862025-04-30T06:38:37.955ZDEBGtest_quotas (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
102872025-04-30T06:38:37.955ZINFOtest_quotas (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
102882025-04-30T06:38:37.955ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = instance_updater elapsed = 194.026\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
102892025-04-30T06:38:37.960ZDEBGtest_quotas (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
102902025-04-30T06:38:37.960ZINFOtest_quotas (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
102912025-04-30T06:38:37.960ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
102922025-04-30T06:38:37.960ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102932025-04-30T06:38:37.960ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
102942025-04-30T06:38:37.960ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
102952025-04-30T06:38:37.961ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
102962025-04-30T06:38:37.961ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
102972025-04-30T06:38:37.970ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
102982025-04-30T06:38:37.972ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 22.450261ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
102992025-04-30T06:38:37.974ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: fe 83 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:60096 req_id = 230d40a5-1379-43d7-8561-6dfe01103461
103002025-04-30T06:38:37.974ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65155,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60096 req_id = 230d40a5-1379-43d7-8561-6dfe01103461
103012025-04-30T06:38:37.974ZDEBGtest_quotas (store): query key key = _dendrite._tcp
103022025-04-30T06:38:37.975ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103032025-04-30T06:38:37.975ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60096 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 230d40a5-1379-43d7-8561-6dfe01103461
103042025-04-30T06:38:37.975ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103052025-04-30T06:38:37.981ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
103062025-04-30T06:38:37.981ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
103072025-04-30T06:38:37.981ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
103082025-04-30T06:38:37.982ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
103092025-04-30T06:38:37.982ZDEBGtest_quotas (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103102025-04-30T06:38:37.986ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
103112025-04-30T06:38:37.986ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
103122025-04-30T06:38:37.986ZDEBGtest_quotas (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103132025-04-30T06:38:37.986ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103142025-04-30T06:38:37.989ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3982, tv_nsec: 788687990 } })
103152025-04-30T06:38:37.989ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
103162025-04-30T06:38:37.991ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 7b 30 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 {0..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62833 req_id = 3210eea7-0324-4492-847a-254381bf49e0
103172025-04-30T06:38:37.991ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31536,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62833 req_id = 3210eea7-0324-4492-847a-254381bf49e0
103182025-04-30T06:38:37.991ZDEBGtest_quotas (store): query key key = _dendrite._tcp
103192025-04-30T06:38:37.991ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103202025-04-30T06:38:37.991ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62833 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3210eea7-0324-4492-847a-254381bf49e0
103212025-04-30T06:38:37.993ZDEBGtest_quotas (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103222025-04-30T06:38:38.006ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
103232025-04-30T06:38:38.006ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
103242025-04-30T06:38:38.006ZDEBGtest_quotas (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
103252025-04-30T06:38:38.006ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
103262025-04-30T06:38:38.007ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103272025-04-30T06:38:38.011ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
103282025-04-30T06:38:38.011ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
103292025-04-30T06:38:38.011ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103302025-04-30T06:38:38.011ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
103312025-04-30T06:38:38.011ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103322025-04-30T06:38:38.012ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
103332025-04-30T06:38:38.014ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
103342025-04-30T06:38:38.014ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
103352025-04-30T06:38:38.014ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103362025-04-30T06:38:38.014ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
103372025-04-30T06:38:38.014ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
103382025-04-30T06:38:38.015ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
103392025-04-30T06:38:38.017ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
103402025-04-30T06:38:38.018ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103412025-04-30T06:38:38.018ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103422025-04-30T06:38:38.018ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103432025-04-30T06:38:38.019ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
103442025-04-30T06:38:38.019ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103452025-04-30T06:38:38.059ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3982, tv_nsec: 858343694 } })
103462025-04-30T06:38:38.059ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
103472025-04-30T06:38:38.060ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103482025-04-30T06:38:38.078ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
103492025-04-30T06:38:38.081ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
103502025-04-30T06:38:38.081ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103512025-04-30T06:38:38.082ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
103522025-04-30T06:38:38.089ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3c 83 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e <...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51408 req_id = 2a2ef154-ddb5-4c94-8c33-cb0f4b393586
103532025-04-30T06:38:38.089ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15491,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51408 req_id = 2a2ef154-ddb5-4c94-8c33-cb0f4b393586
103542025-04-30T06:38:38.089ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103552025-04-30T06:38:38.089ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51408 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2a2ef154-ddb5-4c94-8c33-cb0f4b393586
103562025-04-30T06:38:38.089ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 4a 6e 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 Jn..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:57022 req_id = 32baca81-e174-4524-9045-a5ef985b2559
103572025-04-30T06:38:38.089ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19054,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57022 req_id = 32baca81-e174-4524-9045-a5ef985b2559
103582025-04-30T06:38:38.090ZDEBGtest_quotas (store): query key key = _dendrite._tcp
103592025-04-30T06:38:38.090ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103602025-04-30T06:38:38.090ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57022 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 32baca81-e174-4524-9045-a5ef985b2559
103612025-04-30T06:38:38.091ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
103622025-04-30T06:38:38.210ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
103632025-04-30T06:38:38.210ZDEBGtest_quotas (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
103642025-04-30T06:38:38.210ZDEBGtest_quotas (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
103652025-04-30T06:38:38.210ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
103662025-04-30T06:38:38.211ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
103672025-04-30T06:38:38.212ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3983, tv_nsec: 11126364 } })
103682025-04-30T06:38:38.212ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
103692025-04-30T06:38:38.212ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: de 5b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .[..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63836 req_id = c0859272-f2a2-4c19-9334-7f0299f28747
103702025-04-30T06:38:38.212ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56923,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63836 req_id = c0859272-f2a2-4c19-9334-7f0299f28747
103712025-04-30T06:38:38.212ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
103722025-04-30T06:38:38.212ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63836 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c0859272-f2a2-4c19-9334-7f0299f28747
103732025-04-30T06:38:38.213ZINFOtest_quotas (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
103742025-04-30T06:38:38.213ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 259.18007ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
103752025-04-30T06:38:38.214ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
103762025-04-30T06:38:38.214ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103772025-04-30T06:38:38.214ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
103782025-04-30T06:38:38.214ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
103792025-04-30T06:38:38.216ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103802025-04-30T06:38:38.223ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
103812025-04-30T06:38:38.338ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
103822025-04-30T06:38:38.338ZDEBGtest_quotas (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
103832025-04-30T06:38:38.339ZDEBGtest_quotas (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
103842025-04-30T06:38:38.339ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
103852025-04-30T06:38:38.341ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
103862025-04-30T06:38:38.342ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103872025-04-30T06:38:38.342ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103882025-04-30T06:38:38.345ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
103892025-04-30T06:38:38.346ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
103902025-04-30T06:38:38.470ZINFOtest_quotas (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
103912025-04-30T06:38:38.470ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:55655/vpc-routes
103922025-04-30T06:38:38.472ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103932025-04-30T06:38:38.504ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
103942025-04-30T06:38:38.504ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
103952025-04-30T06:38:38.504ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103962025-04-30T06:38:38.504ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
103972025-04-30T06:38:38.504ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
103982025-04-30T06:38:38.508ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103992025-04-30T06:38:38.511ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104002025-04-30T06:38:38.548ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104012025-04-30T06:38:38.548ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104022025-04-30T06:38:38.549ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
104032025-04-30T06:38:38.550ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104042025-04-30T06:38:38.556ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:62094 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
104052025-04-30T06:38:38.568ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7a 71 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e zq..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54206 req_id = 2384abcb-f9b3-44b2-89e4-062242d6536d
104062025-04-30T06:38:38.568ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31345,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54206 req_id = 2384abcb-f9b3-44b2-89e4-062242d6536d
104072025-04-30T06:38:38.569ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104082025-04-30T06:38:38.569ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54206 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2384abcb-f9b3-44b2-89e4-062242d6536d
104092025-04-30T06:38:38.569ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
104102025-04-30T06:38:38.570ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104112025-04-30T06:38:38.576ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
104122025-04-30T06:38:38.577ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
104132025-04-30T06:38:38.577ZDEBGtest_quotas (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
104142025-04-30T06:38:38.577ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
104152025-04-30T06:38:38.592ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
104162025-04-30T06:38:38.592ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
104172025-04-30T06:38:38.595ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
104182025-04-30T06:38:38.710ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
104192025-04-30T06:38:38.710ZDEBGtest_quotas (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
104202025-04-30T06:38:38.710ZDEBGtest_quotas (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
104212025-04-30T06:38:38.711ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
104222025-04-30T06:38:38.712ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
104232025-04-30T06:38:38.712ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:62094 req_id = c456509c-fe69-42a8-b847-448a0f619dea sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
104242025-04-30T06:38:38.716ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e6f1389-aceb-4c6d-bf6c-de2ca4abec17", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
104252025-04-30T06:38:38.716ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
104262025-04-30T06:38:38.717ZINFOtest_quotas (dropshot): request completed latency_us = 4943 local_addr = [::1]:55655 method = GET remote_addr = [::1]:62094 req_id = c456509c-fe69-42a8-b847-448a0f619dea response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
104272025-04-30T06:38:38.784ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:55655/v2p
104282025-04-30T06:38:38.787ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104292025-04-30T06:38:38.788ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
104302025-04-30T06:38:38.789ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104312025-04-30T06:38:38.790ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:34004 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
104322025-04-30T06:38:38.791ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:55655/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c456509c-fe69-42a8-b847-448a0f619dea", "content-length": "2", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
104332025-04-30T06:38:38.791ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
104342025-04-30T06:38:38.791ZDEBGtest_quotas (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
104352025-04-30T06:38:38.793ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
104362025-04-30T06:38:38.813ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
104372025-04-30T06:38:38.813ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
104382025-04-30T06:38:38.813ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:34004 req_id = bcda78e3-7eb5-4b50-a413-7112a5fe9c39 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
104392025-04-30T06:38:38.815ZINFOtest_quotas (dropshot): request completed latency_us = 2192 local_addr = [::1]:55655 method = GET remote_addr = [::1]:34004 req_id = bcda78e3-7eb5-4b50-a413-7112a5fe9c39 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
104402025-04-30T06:38:38.816ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:55655/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcda78e3-7eb5-4b50-a413-7112a5fe9c39", "content-length": "2", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
104412025-04-30T06:38:38.816ZINFOtest_quotas (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
104422025-04-30T06:38:38.816ZINFOtest_quotas (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
104432025-04-30T06:38:38.816ZINFOtest_quotas (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
104442025-04-30T06:38:38.818ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 866.178785ms iteration = 2 status = {}
104452025-04-30T06:38:38.819ZINFOtest_quotas (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
104462025-04-30T06:38:38.819ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:55655/eip-gateways
104472025-04-30T06:38:38.820ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = PUT remote_addr = [::1]:62094 req_id = 04933e03-cea9-41b1-a68c-782c9526981c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
104482025-04-30T06:38:38.820ZINFOtest_quotas (dropshot): request completed latency_us = 356 local_addr = [::1]:55655 method = PUT remote_addr = [::1]:62094 req_id = 04933e03-cea9-41b1-a68c-782c9526981c response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
104492025-04-30T06:38:38.821ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:55655/eip-gateways", status: 204, headers: {"x-request-id": "04933e03-cea9-41b1-a68c-782c9526981c", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
104502025-04-30T06:38:38.825ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 865.03172ms iteration = 2 status = {}
104512025-04-30T06:38:38.830ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d1ebdc8-326d-459e-8280-90510c1f3e78", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
104522025-04-30T06:38:38.831ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
104532025-04-30T06:38:38.831ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
104542025-04-30T06:38:38.832ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
104552025-04-30T06:38:38.832ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
104562025-04-30T06:38:38.834ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42dbfd89-1979-46f2-8e7e-21d5d3be5965", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
104572025-04-30T06:38:38.834ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
104582025-04-30T06:38:38.836ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a47850dd-7c59-4ff2-957f-eef17bb4aa17", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
104592025-04-30T06:38:38.836ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
104602025-04-30T06:38:38.837ZTRACtest_quotas (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
104612025-04-30T06:38:38.837ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104622025-04-30T06:38:38.837ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
104632025-04-30T06:38:38.837ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
104642025-04-30T06:38:38.838ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
104652025-04-30T06:38:38.843ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29629a5d-d20a-4a3c-b390-8198b6d40a18", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:37 GMT"} })
104662025-04-30T06:38:38.844ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
104672025-04-30T06:38:38.844ZDEBGtest_quotas (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104682025-04-30T06:38:38.851ZDEBGtest_quotas (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
104692025-04-30T06:38:38.851ZERROtest_quotas (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
104702025-04-30T06:38:38.851ZERROtest_quotas (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
104712025-04-30T06:38:38.851ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 900.928922ms iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
104722025-04-30T06:38:38.853ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104732025-04-30T06:38:38.853ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104742025-04-30T06:38:38.868ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104752025-04-30T06:38:38.870ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104762025-04-30T06:38:38.870ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104772025-04-30T06:38:38.870ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
104782025-04-30T06:38:38.870ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2871db8a-e626-47b6-aaa6-64424d171212", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
104792025-04-30T06:38:38.871ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
104802025-04-30T06:38:38.876ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54710ef2-6eb6-40aa-ba73-cd64c4b450fa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
104812025-04-30T06:38:38.876ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
104822025-04-30T06:38:38.879ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdbc323c-ca35-411b-b8e8-164acba5402b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
104832025-04-30T06:38:38.879ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
104842025-04-30T06:38:38.882ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
104852025-04-30T06:38:38.884ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "703759e7-03f6-4c2a-a940-7d035150317e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
104862025-04-30T06:38:38.884ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
104872025-04-30T06:38:38.887ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "403ecd56-8239-4ff3-9242-74f491976547", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
104882025-04-30T06:38:38.887ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
104892025-04-30T06:38:38.890ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa90b3b4-21ba-43cf-8fb7-a87b8fbb21c1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
104902025-04-30T06:38:38.890ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
104912025-04-30T06:38:38.892ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d83e39f2-6043-4d5a-a7f3-34453227d9f4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
104922025-04-30T06:38:38.892ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
104932025-04-30T06:38:38.894ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a374b046-a30d-4f91-a8ba-31bf4be2f09f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
104942025-04-30T06:38:38.894ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
104952025-04-30T06:38:38.895ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61a67b04-a4f6-4ea9-b502-54f76718a9c7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
104962025-04-30T06:38:38.895ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
104972025-04-30T06:38:38.903ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "382e97f5-41aa-4557-86d4-8f86c9e1484e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
104982025-04-30T06:38:38.903ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
104992025-04-30T06:38:38.906ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c65824af-c596-4b3a-a4f8-ac2e696e54a1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105002025-04-30T06:38:38.906ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
105012025-04-30T06:38:38.907ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55544ae2-80a7-45dd-a01a-6ba8e9fe4910", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105022025-04-30T06:38:38.907ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
105032025-04-30T06:38:38.911ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6006a133-cb08-4f47-8ac0-709cc6c43827", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105042025-04-30T06:38:38.911ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
105052025-04-30T06:38:38.914ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e36ca39c-7ada-46ce-a610-b303313ca85a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105062025-04-30T06:38:38.914ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
105072025-04-30T06:38:38.930ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "081de205-b056-4e37-873f-5ab7d342747a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105082025-04-30T06:38:38.931ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
105092025-04-30T06:38:38.936ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105102025-04-30T06:38:38.936ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105112025-04-30T06:38:38.936ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105122025-04-30T06:38:38.936ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
105132025-04-30T06:38:38.937ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105142025-04-30T06:38:38.945ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105152025-04-30T06:38:38.951ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105162025-04-30T06:38:38.951ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105172025-04-30T06:38:38.951ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
105182025-04-30T06:38:38.951ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105192025-04-30T06:38:38.960ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105202025-04-30T06:38:38.960ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
105212025-04-30T06:38:38.960ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105222025-04-30T06:38:38.974ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "551ce525-6e40-4066-a41e-51afc2fe38ab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105232025-04-30T06:38:38.974ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
105242025-04-30T06:38:38.976ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "262a21fa-a646-4876-896d-6d87639a1de2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105252025-04-30T06:38:38.977ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
105262025-04-30T06:38:38.979ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "563d30f5-d070-4367-ac32-0afc166d0f72", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105272025-04-30T06:38:38.979ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
105282025-04-30T06:38:38.983ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a85c6893-957c-4b43-93e4-75eea894acc6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105292025-04-30T06:38:38.983ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
105302025-04-30T06:38:38.988ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eda7ff4a-5fad-4389-be18-ff1919d689bf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105312025-04-30T06:38:38.988ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
105322025-04-30T06:38:38.991ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dff7d6c8-c475-47f0-9c78-2b06d9d71221", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105332025-04-30T06:38:38.991ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
105342025-04-30T06:38:38.994ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3987d388-a753-4a82-b8ce-1022cd6e4b80", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105352025-04-30T06:38:38.994ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
105362025-04-30T06:38:38.995ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "557ecc95-e111-4d02-8ae6-c3c336b6f6ce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105372025-04-30T06:38:38.995ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105382025-04-30T06:38:38.995ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105392025-04-30T06:38:38.995ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105402025-04-30T06:38:38.995ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105412025-04-30T06:38:38.996ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105422025-04-30T06:38:39.002ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
105432025-04-30T06:38:39.004ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105442025-04-30T06:38:39.021ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105452025-04-30T06:38:39.021ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105462025-04-30T06:38:39.021ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105472025-04-30T06:38:39.021ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105482025-04-30T06:38:39.021ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105492025-04-30T06:38:39.021ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105502025-04-30T06:38:39.025ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105512025-04-30T06:38:39.030ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105522025-04-30T06:38:39.030ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105532025-04-30T06:38:39.030ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105542025-04-30T06:38:39.031ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105552025-04-30T06:38:39.084ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
105562025-04-30T06:38:39.084ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
105572025-04-30T06:38:39.084ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105582025-04-30T06:38:39.085ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105592025-04-30T06:38:39.085ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105602025-04-30T06:38:39.085ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
105612025-04-30T06:38:39.085ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105622025-04-30T06:38:39.085ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:57311 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
105632025-04-30T06:38:39.086ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:57311 req_id = fe4f2549-0521-47f6-b3bc-34ae529371d9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105642025-04-30T06:38:39.087ZINFOtest_quotas (dropshot): request completed latency_us = 798 local_addr = [::1]:55655 method = GET remote_addr = [::1]:57311 req_id = fe4f2549-0521-47f6-b3bc-34ae529371d9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105652025-04-30T06:38:39.089ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe4f2549-0521-47f6-b3bc-34ae529371d9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
105662025-04-30T06:38:39.091ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105672025-04-30T06:38:39.091ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105682025-04-30T06:38:39.091ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
105692025-04-30T06:38:39.093ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
105702025-04-30T06:38:39.093ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105712025-04-30T06:38:39.093ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105722025-04-30T06:38:39.093ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105732025-04-30T06:38:39.094ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105742025-04-30T06:38:39.094ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105752025-04-30T06:38:39.100ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105762025-04-30T06:38:39.100ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105772025-04-30T06:38:39.100ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
105782025-04-30T06:38:39.101ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105792025-04-30T06:38:39.103ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
105802025-04-30T06:38:39.103ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
105812025-04-30T06:38:39.103ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
105822025-04-30T06:38:39.104ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
105832025-04-30T06:38:39.104ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105842025-04-30T06:38:39.110ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
105852025-04-30T06:38:39.111ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105862025-04-30T06:38:39.111ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105872025-04-30T06:38:39.112ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105882025-04-30T06:38:39.115ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105892025-04-30T06:38:39.115ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
105902025-04-30T06:38:39.116ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105912025-04-30T06:38:39.117ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
105922025-04-30T06:38:39.118ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
105932025-04-30T06:38:39.122ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
105942025-04-30T06:38:39.122ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
105952025-04-30T06:38:39.147ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
105962025-04-30T06:38:39.148ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
105972025-04-30T06:38:39.148ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.384297642s iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
105982025-04-30T06:38:39.148ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
105992025-04-30T06:38:39.148ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
106002025-04-30T06:38:39.148ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106012025-04-30T06:38:39.148ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106022025-04-30T06:38:39.148ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106032025-04-30T06:38:39.149ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106042025-04-30T06:38:39.151ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
106052025-04-30T06:38:39.153ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106062025-04-30T06:38:39.157ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
106072025-04-30T06:38:39.157ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106082025-04-30T06:38:39.157ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106092025-04-30T06:38:39.158ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106102025-04-30T06:38:39.161ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
106112025-04-30T06:38:39.161ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
106122025-04-30T06:38:39.163ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fb 67 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .g..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64255 req_id = c8c11dc2-5f03-4d9e-9719-bf3b6be9ea01
106132025-04-30T06:38:39.163ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64359,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64255 req_id = c8c11dc2-5f03-4d9e-9719-bf3b6be9ea01
106142025-04-30T06:38:39.163ZDEBGtest_quotas (store): query key key = _mgs._tcp
106152025-04-30T06:38:39.163ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106162025-04-30T06:38:39.163ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64255 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c8c11dc2-5f03-4d9e-9719-bf3b6be9ea01
106172025-04-30T06:38:39.164ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3983, tv_nsec: 963363605 } })
106182025-04-30T06:38:39.164ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
106192025-04-30T06:38:39.164ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 09 1f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54946 req_id = f3cd018e-9835-48c7-b985-b6e64cfb1212
106202025-04-30T06:38:39.164ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2335,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54946 req_id = f3cd018e-9835-48c7-b985-b6e64cfb1212
106212025-04-30T06:38:39.165ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106222025-04-30T06:38:39.165ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54946 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f3cd018e-9835-48c7-b985-b6e64cfb1212
106232025-04-30T06:38:39.225ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
106242025-04-30T06:38:39.226ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106252025-04-30T06:38:39.226ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106262025-04-30T06:38:39.226ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106272025-04-30T06:38:39.226ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
106282025-04-30T06:38:39.227ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106292025-04-30T06:38:39.227ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
106302025-04-30T06:38:39.230ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 04 79 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .y..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64912 req_id = e94b2b76-7575-472d-aa50-35b397459ae4
106312025-04-30T06:38:39.230ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1145,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64912 req_id = e94b2b76-7575-472d-aa50-35b397459ae4
106322025-04-30T06:38:39.230ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
106332025-04-30T06:38:39.231ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1145, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64912 req_id = e94b2b76-7575-472d-aa50-35b397459ae4
106342025-04-30T06:38:39.232ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
106352025-04-30T06:38:39.232ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
106362025-04-30T06:38:39.232ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
106372025-04-30T06:38:39.236ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106382025-04-30T06:38:39.242ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106392025-04-30T06:38:39.242ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106402025-04-30T06:38:39.242ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
106412025-04-30T06:38:39.243ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106422025-04-30T06:38:39.243ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "90d3ead6-6ed1-474b-baa1-c1941be21e0b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106432025-04-30T06:38:39.243ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
106442025-04-30T06:38:39.250ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106452025-04-30T06:38:39.250ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
106462025-04-30T06:38:39.251ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106472025-04-30T06:38:39.252ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbe51145-74a2-439c-bf42-49beffd51b7a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106482025-04-30T06:38:39.252ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
106492025-04-30T06:38:39.258ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffa11ce0-306e-4f33-affc-c1857dc5132f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106502025-04-30T06:38:39.258ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
106512025-04-30T06:38:39.266ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23a68e51-301d-4b65-ad18-86b05bc6c643", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106522025-04-30T06:38:39.267ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
106532025-04-30T06:38:39.270ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d5fee65-dfc7-4eca-aa8a-342e5f530265", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106542025-04-30T06:38:39.270ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
106552025-04-30T06:38:39.274ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8822b108-2776-4890-88de-1c8748236dde", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106562025-04-30T06:38:39.274ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
106572025-04-30T06:38:39.277ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ecdbacd-7849-4614-b66b-55fac6a0b8a5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106582025-04-30T06:38:39.277ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
106592025-04-30T06:38:39.280ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2899684d-5eef-4367-8fb0-a03df2d5cb5b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106602025-04-30T06:38:39.280ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
106612025-04-30T06:38:39.282ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "705c445b-b5f9-4e4d-87c6-d7308da22a92", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106622025-04-30T06:38:39.282ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
106632025-04-30T06:38:39.286ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
106642025-04-30T06:38:39.287ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "577f174c-2712-4208-89e0-3803789ea07f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106652025-04-30T06:38:39.287ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
106662025-04-30T06:38:39.289ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddb6760a-b588-4ad2-8653-b8b0131e9256", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106672025-04-30T06:38:39.290ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
106682025-04-30T06:38:39.294ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "474ea70b-07bb-4341-aed8-65a80f5a585d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106692025-04-30T06:38:39.294ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
106702025-04-30T06:38:39.298ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bdf87e1-0f01-4ba4-8941-9f98c8b5c77b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106712025-04-30T06:38:39.298ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
106722025-04-30T06:38:39.301ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ae3cc32-6df0-418b-9228-0d6065ebb1a6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106732025-04-30T06:38:39.301ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
106742025-04-30T06:38:39.306ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12f32b6d-f785-4964-821b-1e682abf1147", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106752025-04-30T06:38:39.306ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
106762025-04-30T06:38:39.309ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de771c95-f955-4f3f-8dc9-e098a4450566", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106772025-04-30T06:38:39.309ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
106782025-04-30T06:38:39.314ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cfaa9db-c6ad-4412-8cc7-06682c199676", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106792025-04-30T06:38:39.314ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
106802025-04-30T06:38:39.317ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45777158-9717-42ae-ae72-fad0894e21cf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106812025-04-30T06:38:39.318ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
106822025-04-30T06:38:39.321ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c548445f-fa48-45cc-abbf-10fac5589143", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106832025-04-30T06:38:39.321ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
106842025-04-30T06:38:39.322ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9953da1-f32b-4b6f-b4c5-45f7c46bbd08", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106852025-04-30T06:38:39.323ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
106862025-04-30T06:38:39.324ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "423fe491-a95a-484e-8323-b3c143fdc167", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106872025-04-30T06:38:39.324ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
106882025-04-30T06:38:39.325ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a88523e-3daa-44ce-97ab-035e78ed8d7b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106892025-04-30T06:38:39.326ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
106902025-04-30T06:38:39.327ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a11f9e01-46b8-4a0f-9134-c40df8551fb5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106912025-04-30T06:38:39.327ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
106922025-04-30T06:38:39.329ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19750c06-8649-4294-92ef-357028a41da0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106932025-04-30T06:38:39.329ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
106942025-04-30T06:38:39.333ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01c6b30c-0453-4e04-b88f-160aae20bf20", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
106952025-04-30T06:38:39.333ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
106962025-04-30T06:38:39.337ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106972025-04-30T06:38:39.337ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106982025-04-30T06:38:39.337ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106992025-04-30T06:38:39.337ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
107002025-04-30T06:38:39.339ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107012025-04-30T06:38:39.341ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76803ff8-ce3d-4a67-95d5-9297a705eed3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107022025-04-30T06:38:39.341ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
107032025-04-30T06:38:39.343ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107042025-04-30T06:38:39.349ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107052025-04-30T06:38:39.349ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107062025-04-30T06:38:39.349ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
107072025-04-30T06:38:39.350ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107082025-04-30T06:38:39.352ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107092025-04-30T06:38:39.352ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
107102025-04-30T06:38:39.352ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107112025-04-30T06:38:39.354ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81809f30-6c9e-42fc-a1ec-c4fd827c1fbc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107122025-04-30T06:38:39.355ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
107132025-04-30T06:38:39.366ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b5ca80e-db7c-4e18-bd02-4e438a8c34f8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107142025-04-30T06:38:39.366ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
107152025-04-30T06:38:39.370ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95bd24a1-431f-41da-ae05-15585ff27888", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107162025-04-30T06:38:39.371ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
107172025-04-30T06:38:39.374ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53945b32-7d7f-4590-8dc2-9e3b0daeefb2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107182025-04-30T06:38:39.374ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
107192025-04-30T06:38:39.376ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "343ea8dd-9478-4388-8673-efc84ebf3bc3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107202025-04-30T06:38:39.376ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
107212025-04-30T06:38:39.377ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "caced6a0-1b7e-4560-8403-7ab00dec1af0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107222025-04-30T06:38:39.378ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
107232025-04-30T06:38:39.379ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80df5314-9fb0-4cc8-87c6-a68476451424", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107242025-04-30T06:38:39.380ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
107252025-04-30T06:38:39.381ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58e5125f-8b7d-46cf-a098-5fe46f8e524d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107262025-04-30T06:38:39.381ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
107272025-04-30T06:38:39.383ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afb48c2d-2ced-4bd4-879c-9b45861a75cb", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107282025-04-30T06:38:39.383ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
107292025-04-30T06:38:39.386ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49792d31-4a6e-4d08-a1e0-da3f585787a0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107302025-04-30T06:38:39.386ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
107312025-04-30T06:38:39.389ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
107322025-04-30T06:38:39.390ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cb0d008-78c8-4972-b3c5-889af3168905", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107332025-04-30T06:38:39.390ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
107342025-04-30T06:38:39.393ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6dd6a02-e2e5-4141-b556-ec3cada19b6d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107352025-04-30T06:38:39.393ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
107362025-04-30T06:38:39.402ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8bfdb1d-2af9-4dfc-9d66-4b67b6bb42db", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107372025-04-30T06:38:39.403ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
107382025-04-30T06:38:39.407ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "608ed36d-3ee5-438f-a029-6a531ad53207", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107392025-04-30T06:38:39.407ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
107402025-04-30T06:38:39.419ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70a8b1e5-2e87-4a44-92f7-9ddd3b49c45b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107412025-04-30T06:38:39.419ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
107422025-04-30T06:38:39.431ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3de30b01-2dca-4309-957f-2e59e1af2f2b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107432025-04-30T06:38:39.431ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
107442025-04-30T06:38:39.434ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b5f2e6d-6c86-461c-bab5-f5027fb57830", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107452025-04-30T06:38:39.434ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
107462025-04-30T06:38:39.437ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eee54d9-143d-4883-901d-9e54620ea9a3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107472025-04-30T06:38:39.437ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
107482025-04-30T06:38:39.439ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a87afe7-02d5-49ea-a13d-89ec43e40cfd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107492025-04-30T06:38:39.439ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107502025-04-30T06:38:39.439ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107512025-04-30T06:38:39.439ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107522025-04-30T06:38:39.439ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107532025-04-30T06:38:39.441ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107542025-04-30T06:38:39.442ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107552025-04-30T06:38:39.442ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107562025-04-30T06:38:39.442ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107572025-04-30T06:38:39.442ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
107582025-04-30T06:38:39.443ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107592025-04-30T06:38:39.446ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107602025-04-30T06:38:39.458ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107612025-04-30T06:38:39.458ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107622025-04-30T06:38:39.458ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
107632025-04-30T06:38:39.459ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107642025-04-30T06:38:39.460ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107652025-04-30T06:38:39.467ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
107662025-04-30T06:38:39.467ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107672025-04-30T06:38:39.467ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107682025-04-30T06:38:39.467ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107692025-04-30T06:38:39.467ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107702025-04-30T06:38:39.467ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107712025-04-30T06:38:39.470ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107722025-04-30T06:38:39.470ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
107732025-04-30T06:38:39.471ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107742025-04-30T06:38:39.473ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107752025-04-30T06:38:39.481ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
107762025-04-30T06:38:39.481ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107772025-04-30T06:38:39.481ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107782025-04-30T06:38:39.483ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107792025-04-30T06:38:39.560ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
107802025-04-30T06:38:39.560ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
107812025-04-30T06:38:39.561ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:51148 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
107822025-04-30T06:38:39.562ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:51148 req_id = 71a551f5-e958-4677-ac05-77b33cf23980 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
107832025-04-30T06:38:39.562ZINFOtest_quotas (dropshot): request completed latency_us = 650 local_addr = [::1]:55655 method = GET remote_addr = [::1]:51148 req_id = 71a551f5-e958-4677-ac05-77b33cf23980 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
107842025-04-30T06:38:39.563ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "71a551f5-e958-4677-ac05-77b33cf23980", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
107852025-04-30T06:38:39.565ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
107862025-04-30T06:38:39.565ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
107872025-04-30T06:38:39.565ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
107882025-04-30T06:38:39.567ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
107892025-04-30T06:38:39.567ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107902025-04-30T06:38:39.567ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107912025-04-30T06:38:39.567ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107922025-04-30T06:38:39.568ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107932025-04-30T06:38:39.570ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107942025-04-30T06:38:39.577ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
107952025-04-30T06:38:39.578ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107962025-04-30T06:38:39.578ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107972025-04-30T06:38:39.580ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107982025-04-30T06:38:39.709ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
107992025-04-30T06:38:39.726ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
108002025-04-30T06:38:39.726ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
108012025-04-30T06:38:39.726ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 577.863505ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
108022025-04-30T06:38:39.726ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
108032025-04-30T06:38:39.726ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
108042025-04-30T06:38:39.726ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108052025-04-30T06:38:39.726ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108062025-04-30T06:38:39.726ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108072025-04-30T06:38:39.727ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108082025-04-30T06:38:39.734ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108092025-04-30T06:38:39.741ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108102025-04-30T06:38:39.743ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108112025-04-30T06:38:39.743ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108122025-04-30T06:38:39.745ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108132025-04-30T06:38:39.745ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
108142025-04-30T06:38:39.745ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108152025-04-30T06:38:39.746ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1c77dac8-5b96-48e6-8bf2-c3e11846464f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108162025-04-30T06:38:39.746ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1c77dac8-5b96-48e6-8bf2-c3e11846464f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108172025-04-30T06:38:39.746ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1c77dac8-5b96-48e6-8bf2-c3e11846464f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108182025-04-30T06:38:39.747ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1c77dac8-5b96-48e6-8bf2-c3e11846464f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108192025-04-30T06:38:39.749ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
108202025-04-30T06:38:39.749ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
108212025-04-30T06:38:39.750ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 74 51 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 tQ..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51459 req_id = 7873a0ba-65d1-4895-90ad-d613395fa882
108222025-04-30T06:38:39.752ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29777,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51459 req_id = 7873a0ba-65d1-4895-90ad-d613395fa882
108232025-04-30T06:38:39.752ZDEBGtest_quotas (store): query key key = _mgs._tcp
108242025-04-30T06:38:39.752ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108252025-04-30T06:38:39.753ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51459 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7873a0ba-65d1-4895-90ad-d613395fa882
108262025-04-30T06:38:39.753ZINFOtest_quotas (dropshot_internal): request completed latency_us = 7229 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1c77dac8-5b96-48e6-8bf2-c3e11846464f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
108272025-04-30T06:38:39.753ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3984, tv_nsec: 552784598 } })
108282025-04-30T06:38:39.753ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
108292025-04-30T06:38:39.754ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c77dac8-5b96-48e6-8bf2-c3e11846464f", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
108302025-04-30T06:38:39.754ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
108312025-04-30T06:38:39.755ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d42d53cd-1879-4059-a82f-294d9abfe9b8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
108322025-04-30T06:38:39.755ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d42d53cd-1879-4059-a82f-294d9abfe9b8 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
108332025-04-30T06:38:39.755ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d42d53cd-1879-4059-a82f-294d9abfe9b8 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
108342025-04-30T06:38:39.756ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d42d53cd-1879-4059-a82f-294d9abfe9b8 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
108352025-04-30T06:38:39.756ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c9 6a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .j..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52041 req_id = d10be439-407e-4f9b-a365-3e614b3c8b4e
108362025-04-30T06:38:39.756ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51562,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52041 req_id = d10be439-407e-4f9b-a365-3e614b3c8b4e
108372025-04-30T06:38:39.756ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108382025-04-30T06:38:39.756ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52041 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d10be439-407e-4f9b-a365-3e614b3c8b4e
108392025-04-30T06:38:39.821ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
108402025-04-30T06:38:39.823ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
108412025-04-30T06:38:39.823ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108422025-04-30T06:38:39.823ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108432025-04-30T06:38:39.823ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
108442025-04-30T06:38:39.824ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108452025-04-30T06:38:39.824ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
108462025-04-30T06:38:39.824ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
108472025-04-30T06:38:39.825ZINFOtest_quotas (dropshot_internal): request completed latency_us = 70733 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d42d53cd-1879-4059-a82f-294d9abfe9b8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
108482025-04-30T06:38:39.826ZTRACtest_quotas (dropshot): incoming request local_addr = 127.0.0.1:48722 method = GET remote_addr = 127.0.0.1:61278 req_id = c3d60306-00dd-4b1d-be5c-c0cc5eae8b23 uri = /4678dd50-d1e0-4228-8533-c4090342ff58
108492025-04-30T06:38:39.826ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3a 99 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c :............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49254 req_id = 5a8e1366-1c96-493b-9717-307e6642b077
108502025-04-30T06:38:39.826ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15001,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49254 req_id = 5a8e1366-1c96-493b-9717-307e6642b077
108512025-04-30T06:38:39.827ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
108522025-04-30T06:38:39.827ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15001, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49254 req_id = 5a8e1366-1c96-493b-9717-307e6642b077
108532025-04-30T06:38:39.827ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "d42d53cd-1879-4059-a82f-294d9abfe9b8", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:38 GMT"} })
108542025-04-30T06:38:39.827ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
108552025-04-30T06:38:39.827ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108562025-04-30T06:38:39.827ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
108572025-04-30T06:38:39.834ZINFOtest_quotas (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 8643 local_addr = 127.0.0.1:48722 method = GET remote_addr = 127.0.0.1:61278 req_id = c3d60306-00dd-4b1d-be5c-c0cc5eae8b23 response_code = 200 uri = /4678dd50-d1e0-4228-8533-c4090342ff58
108582025-04-30T06:38:39.835ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
108592025-04-30T06:38:39.835ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
108602025-04-30T06:38:39.835ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
108612025-04-30T06:38:39.836ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
108622025-04-30T06:38:39.847ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
108632025-04-30T06:38:39.847ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
108642025-04-30T06:38:39.847ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
108652025-04-30T06:38:39.848ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108662025-04-30T06:38:39.860ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
108672025-04-30T06:38:39.860ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108682025-04-30T06:38:39.860ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
108692025-04-30T06:38:39.861ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108702025-04-30T06:38:39.875ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108712025-04-30T06:38:39.875ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108722025-04-30T06:38:39.877ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108732025-04-30T06:38:39.878ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
108742025-04-30T06:38:39.878ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108752025-04-30T06:38:39.891ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108762025-04-30T06:38:39.892ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108772025-04-30T06:38:39.892ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108782025-04-30T06:38:39.892ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108792025-04-30T06:38:39.894ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "35da4e06-1e00-491f-891c-f9b0bb9b4a0d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
108802025-04-30T06:38:39.894ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
108812025-04-30T06:38:39.897ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7233518-001d-46cd-9f1b-4c6f2101d9a9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
108822025-04-30T06:38:39.897ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
108832025-04-30T06:38:39.901ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdf578ad-486c-413e-8c25-57b595bf558a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
108842025-04-30T06:38:39.901ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
108852025-04-30T06:38:39.907ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a95193f-6ff4-4bfd-9eaf-698a02d5d6e3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
108862025-04-30T06:38:39.907ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
108872025-04-30T06:38:39.909ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "930d4905-d3e8-4ce9-afb9-23cc6a68d44d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
108882025-04-30T06:38:39.910ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
108892025-04-30T06:38:39.915ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10f537b5-6dd4-40b7-ba82-609321b4710c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
108902025-04-30T06:38:39.915ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
108912025-04-30T06:38:39.919ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb2e0c5d-3cf0-4353-8617-87e64b419288", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
108922025-04-30T06:38:39.919ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
108932025-04-30T06:38:39.938ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
108942025-04-30T06:38:39.952ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d971b9cd-0f1a-48b0-993a-88315cd1691b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
108952025-04-30T06:38:39.952ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
108962025-04-30T06:38:39.968ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "233ece77-5120-4ff8-ab12-b1ca3f8fe639", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
108972025-04-30T06:38:39.968ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
108982025-04-30T06:38:39.987ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30179575-c7f3-44c5-bb60-e6addba4a089", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
108992025-04-30T06:38:39.988ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
109002025-04-30T06:38:39.989ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab754497-93c7-4fa7-a54c-7f354e7254ad", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109012025-04-30T06:38:39.989ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
109022025-04-30T06:38:39.990ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1507b62-ab0f-4b18-bebf-6cae029632d0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109032025-04-30T06:38:39.990ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
109042025-04-30T06:38:39.991ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
109052025-04-30T06:38:39.991ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109062025-04-30T06:38:39.991ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109072025-04-30T06:38:39.991ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
109082025-04-30T06:38:39.992ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109092025-04-30T06:38:39.993ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30de491c-c896-4083-b47c-4cf8518ae4f2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109102025-04-30T06:38:39.994ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
109112025-04-30T06:38:39.995ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109122025-04-30T06:38:40.008ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
109132025-04-30T06:38:40.009ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109142025-04-30T06:38:40.009ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
109152025-04-30T06:38:40.009ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109162025-04-30T06:38:40.014ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfb54881-ed9a-4935-8eab-35e700bd964b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109172025-04-30T06:38:40.014ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
109182025-04-30T06:38:40.016ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109192025-04-30T06:38:40.016ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
109202025-04-30T06:38:40.017ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109212025-04-30T06:38:40.019ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4789c3f5-6e0a-447b-9027-d52a7335ce76", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109222025-04-30T06:38:40.020ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
109232025-04-30T06:38:40.022ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fb91fbb-49e5-48b5-9d2e-d5e9dbc6a373", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109242025-04-30T06:38:40.022ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
109252025-04-30T06:38:40.024ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d867312b-b64f-4b18-9b9d-61780d63c6e9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109262025-04-30T06:38:40.024ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
109272025-04-30T06:38:40.026ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d89afb5f-953b-4cc2-9eac-3ecd9e82d2a2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109282025-04-30T06:38:40.026ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
109292025-04-30T06:38:40.043ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e432976a-1d9d-4a4c-9764-c35d2b3db806", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109302025-04-30T06:38:40.044ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
109312025-04-30T06:38:40.045ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fa222b1-2467-4f2a-ace8-7d1543043eeb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109322025-04-30T06:38:40.046ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
109332025-04-30T06:38:40.048ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c359ea25-532f-4654-b912-c7a4122cabe2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109342025-04-30T06:38:40.048ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
109352025-04-30T06:38:40.048ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d92a88b3-a2db-4fa3-845d-6bae34d50250", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109362025-04-30T06:38:40.048ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
109372025-04-30T06:38:40.050ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "abd52fed-a1b8-46bf-9f28-2d2bd5991332", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109382025-04-30T06:38:40.050ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
109392025-04-30T06:38:40.052ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c608d842-540d-4672-9530-ea72ff2fc122", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109402025-04-30T06:38:40.053ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
109412025-04-30T06:38:40.059ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
109422025-04-30T06:38:40.060ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ee61ea8-84bd-4650-8521-214035147fea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109432025-04-30T06:38:40.060ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
109442025-04-30T06:38:40.062ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eedc99c-0817-4de4-bb0d-fb0615154967", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109452025-04-30T06:38:40.062ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
109462025-04-30T06:38:40.064ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b592a722-7361-43fd-a899-e4c1a3c82892", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109472025-04-30T06:38:40.065ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
109482025-04-30T06:38:40.066ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6a1e6a1-18a2-45b0-8c8b-87b45619391e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109492025-04-30T06:38:40.067ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
109502025-04-30T06:38:40.069ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "855b9e78-d13b-4e06-af52-662d2a9bf69c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109512025-04-30T06:38:40.069ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
109522025-04-30T06:38:40.072ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "baccc1c3-114b-4a74-b8d4-d26ff7926dbb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109532025-04-30T06:38:40.072ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
109542025-04-30T06:38:40.073ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69250717-d6be-49ae-a5f9-877f3af5db77", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109552025-04-30T06:38:40.073ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
109562025-04-30T06:38:40.074ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6ab44fc-c2da-4e05-b95f-02c980184bdc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109572025-04-30T06:38:40.074ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
109582025-04-30T06:38:40.075ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "332c49bf-2af8-4531-a06e-9301fbf3d1bc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109592025-04-30T06:38:40.075ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
109602025-04-30T06:38:40.076ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcd2529a-7f99-45ac-934b-3b30f0b650a1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109612025-04-30T06:38:40.076ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
109622025-04-30T06:38:40.078ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db905120-dd42-4780-b46b-074d1c5fc584", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109632025-04-30T06:38:40.079ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
109642025-04-30T06:38:40.081ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95bfac48-a7da-4530-841a-aabe46c3e389", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109652025-04-30T06:38:40.086ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
109662025-04-30T06:38:40.089ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f502ab3-4e0c-46b0-a7ad-65af24644922", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109672025-04-30T06:38:40.089ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
109682025-04-30T06:38:40.091ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f69d2cf-4860-4940-ae9f-66d64a5b095c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109692025-04-30T06:38:40.091ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
109702025-04-30T06:38:40.093ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1c1c473-3897-43ca-bab3-82606004251e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109712025-04-30T06:38:40.093ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
109722025-04-30T06:38:40.094ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bdfd5be-18c5-4a6f-a052-87b19436a86d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109732025-04-30T06:38:40.095ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
109742025-04-30T06:38:40.096ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4555f32-7142-4197-a6f7-f67d5a9e16a6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109752025-04-30T06:38:40.096ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
109762025-04-30T06:38:40.097ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9166ac1e-4d18-4488-bc5b-50fa21c1dda5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109772025-04-30T06:38:40.098ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
109782025-04-30T06:38:40.099ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10469135-bf59-4610-96f2-1aee7e2241d2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109792025-04-30T06:38:40.099ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
109802025-04-30T06:38:40.100ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db381081-14de-42d8-834a-f4933e3ebae2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109812025-04-30T06:38:40.100ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
109822025-04-30T06:38:40.101ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c974afb-93f4-40a2-9590-c7f29cbfcabc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
109832025-04-30T06:38:40.101ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
109842025-04-30T06:38:40.101ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109852025-04-30T06:38:40.101ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109862025-04-30T06:38:40.101ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109872025-04-30T06:38:40.102ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109882025-04-30T06:38:40.103ZDEBGtest_quotas (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109892025-04-30T06:38:40.103ZDEBGtest_quotas (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109902025-04-30T06:38:40.103ZDEBGtest_quotas (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
109912025-04-30T06:38:40.109ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
109922025-04-30T06:38:40.110ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
109932025-04-30T06:38:40.110ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109942025-04-30T06:38:40.110ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109952025-04-30T06:38:40.110ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
109962025-04-30T06:38:40.111ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109972025-04-30T06:38:40.112ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109982025-04-30T06:38:40.116ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
109992025-04-30T06:38:40.116ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
110002025-04-30T06:38:40.116ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110012025-04-30T06:38:40.116ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110022025-04-30T06:38:40.116ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110032025-04-30T06:38:40.117ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110042025-04-30T06:38:40.122ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110052025-04-30T06:38:40.128ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
110062025-04-30T06:38:40.128ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110072025-04-30T06:38:40.128ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
110082025-04-30T06:38:40.129ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110092025-04-30T06:38:40.129ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 12 table_name = fields_string
110102025-04-30T06:38:40.129ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
110112025-04-30T06:38:40.132ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110122025-04-30T06:38:40.137ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
110132025-04-30T06:38:40.137ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110142025-04-30T06:38:40.137ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110152025-04-30T06:38:40.138ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110162025-04-30T06:38:40.143ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 5 table_name = fields_u16
110172025-04-30T06:38:40.143ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
110182025-04-30T06:38:40.152ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110192025-04-30T06:38:40.163ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
110202025-04-30T06:38:40.164ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110212025-04-30T06:38:40.286ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
110222025-04-30T06:38:40.287ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
110232025-04-30T06:38:40.292ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:55413 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
110242025-04-30T06:38:40.315ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 5 table_name = fields_uuid
110252025-04-30T06:38:40.315ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
110262025-04-30T06:38:40.317ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
110272025-04-30T06:38:40.318ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
110282025-04-30T06:38:40.324ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:55413 req_id = b0c4fcd1-4996-4f02-9ca8-82d78c05fade sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
110292025-04-30T06:38:40.325ZINFOtest_quotas (dropshot): request completed latency_us = 1095 local_addr = [::1]:55655 method = GET remote_addr = [::1]:55413 req_id = b0c4fcd1-4996-4f02-9ca8-82d78c05fade response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
110302025-04-30T06:38:40.326ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 5 table_name = measurements_histogramu64
110312025-04-30T06:38:40.326ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
110322025-04-30T06:38:40.328ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: e6 c0 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:56125 req_id = 40433d77-2519-4591-ba5b-a3d850408fe9
110332025-04-30T06:38:40.328ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59072,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56125 req_id = 40433d77-2519-4591-ba5b-a3d850408fe9
110342025-04-30T06:38:40.328ZDEBGtest_quotas (store): query key key = _dendrite._tcp
110352025-04-30T06:38:40.330ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110362025-04-30T06:38:40.331ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56125 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 40433d77-2519-4591-ba5b-a3d850408fe9
110372025-04-30T06:38:40.331ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0c4fcd1-4996-4f02-9ca8-82d78c05fade", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:39 GMT"} })
110382025-04-30T06:38:40.357ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
110392025-04-30T06:38:40.358ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
110402025-04-30T06:38:40.358ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
110412025-04-30T06:38:40.367ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
110422025-04-30T06:38:40.367ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110432025-04-30T06:38:40.367ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110442025-04-30T06:38:40.367ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110452025-04-30T06:38:40.368ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110462025-04-30T06:38:40.374ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3985, tv_nsec: 173442197 } })
110472025-04-30T06:38:40.374ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
110482025-04-30T06:38:40.375ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
110492025-04-30T06:38:40.380ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110502025-04-30T06:38:40.384ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
110512025-04-30T06:38:40.385ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110522025-04-30T06:38:40.385ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110532025-04-30T06:38:40.386ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110542025-04-30T06:38:40.387ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
110552025-04-30T06:38:40.387ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
110562025-04-30T06:38:40.391ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: be ad 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59320 req_id = 7d21c59f-dc27-4bc4-9ebc-14947d1f7bfb
110572025-04-30T06:38:40.391ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48813,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59320 req_id = 7d21c59f-dc27-4bc4-9ebc-14947d1f7bfb
110582025-04-30T06:38:40.393ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110592025-04-30T06:38:40.393ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59320 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7d21c59f-dc27-4bc4-9ebc-14947d1f7bfb
110602025-04-30T06:38:40.393ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 88 5f 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ._..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:54429 req_id = aa08cb78-2876-4189-80ad-5c58a740478a
110612025-04-30T06:38:40.393ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34911,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54429 req_id = aa08cb78-2876-4189-80ad-5c58a740478a
110622025-04-30T06:38:40.393ZDEBGtest_quotas (store): query key key = _dendrite._tcp
110632025-04-30T06:38:40.393ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110642025-04-30T06:38:40.393ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54429 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = aa08cb78-2876-4189-80ad-5c58a740478a
110652025-04-30T06:38:40.394ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
110662025-04-30T06:38:40.512ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
110672025-04-30T06:38:40.512ZDEBGtest_quotas (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
110682025-04-30T06:38:40.513ZDEBGtest_quotas (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
110692025-04-30T06:38:40.513ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
110702025-04-30T06:38:40.514ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
110712025-04-30T06:38:40.514ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3985, tv_nsec: 313709120 } })
110722025-04-30T06:38:40.514ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
110732025-04-30T06:38:40.544ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5e 85 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ^...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56896 req_id = b138a2ef-bd6f-4e53-af9e-78ce17585466
110742025-04-30T06:38:40.544ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24197,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56896 req_id = b138a2ef-bd6f-4e53-af9e-78ce17585466
110752025-04-30T06:38:40.545ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110762025-04-30T06:38:40.545ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56896 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b138a2ef-bd6f-4e53-af9e-78ce17585466
110772025-04-30T06:38:40.547ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
110782025-04-30T06:38:40.621ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
110792025-04-30T06:38:40.622ZDEBGtest_quotas (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
110802025-04-30T06:38:40.622ZDEBGtest_quotas (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
110812025-04-30T06:38:40.622ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
110822025-04-30T06:38:40.625ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
110832025-04-30T06:38:40.644ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
110842025-04-30T06:38:40.690ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
110852025-04-30T06:38:40.691ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
110862025-04-30T06:38:40.691ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 964.610503ms iteration = 38 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
110872025-04-30T06:38:40.693ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 39 reason = Signaled
110882025-04-30T06:38:40.693ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
110892025-04-30T06:38:40.693ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110902025-04-30T06:38:40.693ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110912025-04-30T06:38:40.693ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110922025-04-30T06:38:40.694ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110932025-04-30T06:38:40.700ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
110942025-04-30T06:38:40.704ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110952025-04-30T06:38:40.704ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110962025-04-30T06:38:40.704ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
110972025-04-30T06:38:40.705ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110982025-04-30T06:38:40.745ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110992025-04-30T06:38:40.757ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
111002025-04-30T06:38:40.757ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111012025-04-30T06:38:40.757ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111022025-04-30T06:38:40.758ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111032025-04-30T06:38:40.765ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
111042025-04-30T06:38:40.768ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111052025-04-30T06:38:40.784ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111062025-04-30T06:38:40.793ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111072025-04-30T06:38:40.793ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
111082025-04-30T06:38:40.797ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111092025-04-30T06:38:40.799ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
111102025-04-30T06:38:40.799ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
111112025-04-30T06:38:40.802ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 2f f4 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 /............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52885 req_id = 3454b5cc-9aa2-4d1b-85b6-2565abf83dda
111122025-04-30T06:38:40.802ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12276,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52885 req_id = 3454b5cc-9aa2-4d1b-85b6-2565abf83dda
111132025-04-30T06:38:40.802ZDEBGtest_quotas (store): query key key = _dendrite._tcp
111142025-04-30T06:38:40.819ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111152025-04-30T06:38:40.819ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52885 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3454b5cc-9aa2-4d1b-85b6-2565abf83dda
111162025-04-30T06:38:40.819ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111172025-04-30T06:38:40.819ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
111182025-04-30T06:38:40.820ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111192025-04-30T06:38:40.820ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3985, tv_nsec: 619802521 } })
111202025-04-30T06:38:40.820ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
111212025-04-30T06:38:40.821ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3e f4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 >............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49755 req_id = be1071e3-8657-4b60-ab23-f1bbb1185b9c
111222025-04-30T06:38:40.821ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16116,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49755 req_id = be1071e3-8657-4b60-ab23-f1bbb1185b9c
111232025-04-30T06:38:40.821ZDEBGtest_quotas (store): query key key = _mgs._tcp
111242025-04-30T06:38:40.822ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111252025-04-30T06:38:40.830ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49755 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = be1071e3-8657-4b60-ab23-f1bbb1185b9c
111262025-04-30T06:38:40.831ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3985, tv_nsec: 630385467 } })
111272025-04-30T06:38:40.831ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
111282025-04-30T06:38:40.831ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 95 bd 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51058 req_id = 9f9c2e0c-372b-4fbb-8bb1-492fc96beb5d
111292025-04-30T06:38:40.831ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38333,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51058 req_id = 9f9c2e0c-372b-4fbb-8bb1-492fc96beb5d
111302025-04-30T06:38:40.832ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111312025-04-30T06:38:40.832ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51058 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9f9c2e0c-372b-4fbb-8bb1-492fc96beb5d
111322025-04-30T06:38:40.833ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
111332025-04-30T06:38:41.024ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
111342025-04-30T06:38:41.024ZDEBGtest_quotas (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
111352025-04-30T06:38:41.025ZDEBGtest_quotas (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
111362025-04-30T06:38:41.025ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
111372025-04-30T06:38:41.026ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
111382025-04-30T06:38:41.026ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a9 30 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .0..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53632 req_id = cae6c557-402f-45dd-943e-bb116c658958
111392025-04-30T06:38:41.026ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43312,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53632 req_id = cae6c557-402f-45dd-943e-bb116c658958
111402025-04-30T06:38:41.027ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111412025-04-30T06:38:41.027ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53632 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cae6c557-402f-45dd-943e-bb116c658958
111422025-04-30T06:38:41.110ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
111432025-04-30T06:38:41.110ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111442025-04-30T06:38:41.112ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111452025-04-30T06:38:41.119ZTRACtest_quotas (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111462025-04-30T06:38:41.119ZTRACtest_quotas (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111472025-04-30T06:38:41.122ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 98 b4 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49654 req_id = 698bde53-8895-4b62-826a-21c21cf785b2
111482025-04-30T06:38:41.122ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39092,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49654 req_id = 698bde53-8895-4b62-826a-21c21cf785b2
111492025-04-30T06:38:41.122ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
111502025-04-30T06:38:41.123ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39092, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49654 req_id = 698bde53-8895-4b62-826a-21c21cf785b2
111512025-04-30T06:38:41.123ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111522025-04-30T06:38:41.124ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111532025-04-30T06:38:41.124ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111542025-04-30T06:38:41.124ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111552025-04-30T06:38:41.124ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
111562025-04-30T06:38:41.125ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
111572025-04-30T06:38:41.126ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
111582025-04-30T06:38:41.136ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "94a64af0-0449-47e9-b927-17380d34435e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111592025-04-30T06:38:41.137ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
111602025-04-30T06:38:41.153ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "619760b3-ce61-45e7-942d-8e529cb17ad2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111612025-04-30T06:38:41.153ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
111622025-04-30T06:38:41.239ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fae2774c-0141-4070-9ad0-edba40cd20b3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111632025-04-30T06:38:41.239ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
111642025-04-30T06:38:41.241ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41b5bd8c-f364-455b-911b-4162153ca9fe", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111652025-04-30T06:38:41.241ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
111662025-04-30T06:38:41.251ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "725c2cb6-ff45-4ead-ad1c-996bed50a90f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111672025-04-30T06:38:41.251ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
111682025-04-30T06:38:41.251ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
111692025-04-30T06:38:41.265ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "def648a4-faa9-47b9-a458-07f10c750920", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111702025-04-30T06:38:41.266ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
111712025-04-30T06:38:41.269ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b082886-b24c-4017-a2a2-dc0c62bdbeb0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111722025-04-30T06:38:41.269ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
111732025-04-30T06:38:41.272ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "937d6f65-9fbb-458f-aa25-c1419260957e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111742025-04-30T06:38:41.272ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
111752025-04-30T06:38:41.274ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddcd5276-257e-4de7-b352-1b318a115385", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111762025-04-30T06:38:41.274ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
111772025-04-30T06:38:41.299ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5973e548-7991-4694-9943-744aacda60ef", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111782025-04-30T06:38:41.316ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
111792025-04-30T06:38:41.317ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
111802025-04-30T06:38:41.317ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111812025-04-30T06:38:41.317ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111822025-04-30T06:38:41.317ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
111832025-04-30T06:38:41.318ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111842025-04-30T06:38:41.318ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc42e31b-e408-42f0-8abd-1f28eb043ddd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111852025-04-30T06:38:41.319ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
111862025-04-30T06:38:41.320ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdee3237-4846-4dbb-8c0b-fac3e077799e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111872025-04-30T06:38:41.320ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
111882025-04-30T06:38:41.321ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a7f84e5-653b-40d2-907c-66716e93dfef", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111892025-04-30T06:38:41.321ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
111902025-04-30T06:38:41.322ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111912025-04-30T06:38:41.337ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111922025-04-30T06:38:41.337ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111932025-04-30T06:38:41.337ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
111942025-04-30T06:38:41.338ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111952025-04-30T06:38:41.340ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d757fb8c-da9c-4f67-bc2c-11a69fbb6981", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
111962025-04-30T06:38:41.340ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
111972025-04-30T06:38:41.340ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111982025-04-30T06:38:41.340ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
111992025-04-30T06:38:41.341ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112002025-04-30T06:38:41.343ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81d0ab10-231b-47d4-bb37-7866d4019762", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112012025-04-30T06:38:41.343ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
112022025-04-30T06:38:41.397ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93bd898c-5ef4-47f3-bea7-6c2533031d67", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112032025-04-30T06:38:41.397ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
112042025-04-30T06:38:41.410ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02a46283-1bdf-4f0f-8238-2805c190734b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112052025-04-30T06:38:41.410ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
112062025-04-30T06:38:41.419ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3429c7c0-2d22-40d9-a295-12551391911b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112072025-04-30T06:38:41.419ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
112082025-04-30T06:38:41.420ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
112092025-04-30T06:38:41.445ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d71db0e-09e0-47a8-be0e-d5518add687d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112102025-04-30T06:38:41.445ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
112112025-04-30T06:38:41.446ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "665ccbbc-a3b8-489d-87be-2572dfebe297", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112122025-04-30T06:38:41.447ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
112132025-04-30T06:38:41.450ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b67708ce-8b81-463e-a963-7a2a5b0e6fe9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112142025-04-30T06:38:41.452ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
112152025-04-30T06:38:41.463ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8aac2c1-8bb3-4bcc-b352-3b1461957383", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112162025-04-30T06:38:41.463ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
112172025-04-30T06:38:41.465ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96661276-50d3-4a4d-a8b5-b9416cfc7f55", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112182025-04-30T06:38:41.465ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
112192025-04-30T06:38:41.467ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97385b4f-68df-47e8-a164-23081c35edce", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112202025-04-30T06:38:41.467ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
112212025-04-30T06:38:41.471ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112222025-04-30T06:38:41.471ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112232025-04-30T06:38:41.471ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112242025-04-30T06:38:41.471ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
112252025-04-30T06:38:41.473ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112262025-04-30T06:38:41.491ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c254fc96-6e2f-4c7c-8ff3-486f4a3f0be9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112272025-04-30T06:38:41.497ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
112282025-04-30T06:38:41.497ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112292025-04-30T06:38:41.518ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112302025-04-30T06:38:41.518ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112312025-04-30T06:38:41.518ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
112322025-04-30T06:38:41.521ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112332025-04-30T06:38:41.524ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112342025-04-30T06:38:41.524ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
112352025-04-30T06:38:41.525ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112362025-04-30T06:38:41.528ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0f6a7ff-4ae1-49c9-963e-3829e8f013d8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112372025-04-30T06:38:41.528ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
112382025-04-30T06:38:41.594ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4946f04d-c54f-4f5e-bf1d-be39574a1064", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112392025-04-30T06:38:41.594ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
112402025-04-30T06:38:41.604ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56fbfdf1-add1-4f30-84b5-b2dfe5b2a50c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112412025-04-30T06:38:41.604ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
112422025-04-30T06:38:41.638ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d9ece05-7cd7-4dfd-8d98-1f62256e450e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112432025-04-30T06:38:41.638ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
112442025-04-30T06:38:41.646ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6de2c0c8-5283-4ac9-963c-bb89e6fa5664", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112452025-04-30T06:38:41.646ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
112462025-04-30T06:38:41.647ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
112472025-04-30T06:38:41.648ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "322b9a23-40cf-4c50-9910-72bcafe2df0b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112482025-04-30T06:38:41.648ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
112492025-04-30T06:38:41.650ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ad16ccf-2ae3-4c78-b1d5-d59a02407130", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112502025-04-30T06:38:41.651ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
112512025-04-30T06:38:41.652ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e551e82d-d24c-48ed-9861-64ff426e4d33", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112522025-04-30T06:38:41.652ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
112532025-04-30T06:38:41.654ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "019da675-8ccf-4058-8908-2b24d72d3cfb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112542025-04-30T06:38:41.654ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
112552025-04-30T06:38:41.656ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e183f330-c104-4731-8e9e-fd5138730c47", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112562025-04-30T06:38:41.657ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
112572025-04-30T06:38:41.660ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb57e5d0-51d2-443c-9d70-4c65ea6c8c26", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112582025-04-30T06:38:41.667ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
112592025-04-30T06:38:41.667ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71f9d87c-58d6-4ba7-b3e6-4122b8474505", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112602025-04-30T06:38:41.667ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
112612025-04-30T06:38:41.667ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5c6b799-d7a7-4aee-ab0f-f9c06c4b71e0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112622025-04-30T06:38:41.667ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
112632025-04-30T06:38:41.668ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4dbacd9-4a25-496c-a31c-cc96c1efa0bf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112642025-04-30T06:38:41.668ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
112652025-04-30T06:38:41.671ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69e20729-2b7c-4d72-a565-15e11d600434", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112662025-04-30T06:38:41.672ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
112672025-04-30T06:38:41.675ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f59f9fd1-f8fb-4563-9f55-8d10a40b45c2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112682025-04-30T06:38:41.675ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
112692025-04-30T06:38:41.676ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "037a330c-bf6b-4bef-ba4e-80b1c63e8b60", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112702025-04-30T06:38:41.676ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
112712025-04-30T06:38:41.677ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7e81891-b8e1-4f84-b054-b599222d5b06", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112722025-04-30T06:38:41.677ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
112732025-04-30T06:38:41.678ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c46f3cd6-0084-4e27-8c21-d19e3c966834", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112742025-04-30T06:38:41.678ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
112752025-04-30T06:38:41.680ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a36f78c8-110a-4f42-ab5f-ea13b2bf423d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
112762025-04-30T06:38:41.680ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112772025-04-30T06:38:41.680ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112782025-04-30T06:38:41.680ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112792025-04-30T06:38:41.680ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112802025-04-30T06:38:41.681ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112812025-04-30T06:38:41.752ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112822025-04-30T06:38:41.753ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112832025-04-30T06:38:41.753ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112842025-04-30T06:38:41.753ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
112852025-04-30T06:38:41.756ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112862025-04-30T06:38:41.757ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112872025-04-30T06:38:41.766ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112882025-04-30T06:38:41.766ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112892025-04-30T06:38:41.766ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112902025-04-30T06:38:41.766ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112912025-04-30T06:38:41.766ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112922025-04-30T06:38:41.767ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112932025-04-30T06:38:41.767ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
112942025-04-30T06:38:41.767ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
112952025-04-30T06:38:41.769ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1c4b44da-a406-4cda-8caa-ff623808a293 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
112962025-04-30T06:38:41.769ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1c4b44da-a406-4cda-8caa-ff623808a293 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
112972025-04-30T06:38:41.769ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1c4b44da-a406-4cda-8caa-ff623808a293 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
112982025-04-30T06:38:41.770ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1c4b44da-a406-4cda-8caa-ff623808a293 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
112992025-04-30T06:38:41.771ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113002025-04-30T06:38:41.778ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113012025-04-30T06:38:41.779ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113022025-04-30T06:38:41.779ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
113032025-04-30T06:38:41.779ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113042025-04-30T06:38:41.780ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113052025-04-30T06:38:41.784ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
113062025-04-30T06:38:41.784ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113072025-04-30T06:38:41.784ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113082025-04-30T06:38:41.785ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113092025-04-30T06:38:41.786ZINFOtest_quotas (dropshot_internal): request completed latency_us = 16654 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 1c4b44da-a406-4cda-8caa-ff623808a293 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
113102025-04-30T06:38:41.786ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c4b44da-a406-4cda-8caa-ff623808a293", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
113112025-04-30T06:38:41.786ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
113122025-04-30T06:38:41.787ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113132025-04-30T06:38:41.787ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
113142025-04-30T06:38:41.787ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113152025-04-30T06:38:41.788ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 5ba28775-d0f3-4303-9bca-603f8e153b36 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
113162025-04-30T06:38:41.788ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 5ba28775-d0f3-4303-9bca-603f8e153b36 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
113172025-04-30T06:38:41.788ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 5ba28775-d0f3-4303-9bca-603f8e153b36 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
113182025-04-30T06:38:41.789ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 5ba28775-d0f3-4303-9bca-603f8e153b36 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
113192025-04-30T06:38:41.897ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
113202025-04-30T06:38:41.897ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
113212025-04-30T06:38:41.899ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:64126 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113222025-04-30T06:38:41.899ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113232025-04-30T06:38:41.899ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113242025-04-30T06:38:41.900ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:64126 req_id = b2f10773-e8a4-4d39-99f2-cb1ea1f3ab24 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113252025-04-30T06:38:41.905ZINFOtest_quotas (dropshot): request completed latency_us = 2401 local_addr = [::1]:55655 method = GET remote_addr = [::1]:64126 req_id = b2f10773-e8a4-4d39-99f2-cb1ea1f3ab24 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113262025-04-30T06:38:41.908ZINFOtest_quotas (dropshot_internal): request completed latency_us = 119936 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 5ba28775-d0f3-4303-9bca-603f8e153b36 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
113272025-04-30T06:38:41.909ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ba28775-d0f3-4303-9bca-603f8e153b36", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
113282025-04-30T06:38:41.909ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
113292025-04-30T06:38:41.909ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113302025-04-30T06:38:41.909ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
113312025-04-30T06:38:41.909ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113322025-04-30T06:38:41.909ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113332025-04-30T06:38:41.909ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113342025-04-30T06:38:41.910ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113352025-04-30T06:38:41.910ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2f10773-e8a4-4d39-99f2-cb1ea1f3ab24", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:41 GMT"} })
113362025-04-30T06:38:41.912ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113372025-04-30T06:38:41.912ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113382025-04-30T06:38:41.912ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
113392025-04-30T06:38:41.914ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
113402025-04-30T06:38:41.914ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113412025-04-30T06:38:41.914ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113422025-04-30T06:38:41.914ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113432025-04-30T06:38:41.915ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113442025-04-30T06:38:41.920ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113452025-04-30T06:38:41.925ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
113462025-04-30T06:38:41.925ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113472025-04-30T06:38:41.925ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113482025-04-30T06:38:41.926ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113492025-04-30T06:38:41.981ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
113502025-04-30T06:38:41.999ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
113512025-04-30T06:38:41.999ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
113522025-04-30T06:38:41.999ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.306696046s iteration = 39 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
113532025-04-30T06:38:42.000ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 40 reason = Signaled
113542025-04-30T06:38:42.000ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
113552025-04-30T06:38:42.001ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113562025-04-30T06:38:42.001ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113572025-04-30T06:38:42.001ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113582025-04-30T06:38:42.001ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113592025-04-30T06:38:42.009ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113602025-04-30T06:38:42.016ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
113612025-04-30T06:38:42.016ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113622025-04-30T06:38:42.016ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113632025-04-30T06:38:42.017ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113642025-04-30T06:38:42.020ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
113652025-04-30T06:38:42.020ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
113662025-04-30T06:38:42.035ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 65 6c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 el..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50903 req_id = 8b7e905a-0b9c-4b31-9500-ced6da55968d
113672025-04-30T06:38:42.035ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25964,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50903 req_id = 8b7e905a-0b9c-4b31-9500-ced6da55968d
113682025-04-30T06:38:42.035ZDEBGtest_quotas (store): query key key = _mgs._tcp
113692025-04-30T06:38:42.035ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113702025-04-30T06:38:42.035ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50903 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8b7e905a-0b9c-4b31-9500-ced6da55968d
113712025-04-30T06:38:42.035ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113722025-04-30T06:38:42.036ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113732025-04-30T06:38:42.036ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113742025-04-30T06:38:42.036ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
113752025-04-30T06:38:42.036ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113762025-04-30T06:38:42.037ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3986, tv_nsec: 835987771 } })
113772025-04-30T06:38:42.037ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
113782025-04-30T06:38:42.038ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f9 b1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54317 req_id = 0d8a2957-8cc2-4ac0-8ba6-155dd4145537
113792025-04-30T06:38:42.038ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63921,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54317 req_id = 0d8a2957-8cc2-4ac0-8ba6-155dd4145537
113802025-04-30T06:38:42.038ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113812025-04-30T06:38:42.039ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54317 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0d8a2957-8cc2-4ac0-8ba6-155dd4145537
113822025-04-30T06:38:42.098ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
113832025-04-30T06:38:42.099ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113842025-04-30T06:38:42.106ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113852025-04-30T06:38:42.106ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113862025-04-30T06:38:42.106ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
113872025-04-30T06:38:42.107ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113882025-04-30T06:38:42.119ZDEBGtest_quotas (results-sink-cluster): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
113892025-04-30T06:38:42.119ZDEBGtest_quotas (results-sink): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
113902025-04-30T06:38:42.119ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6f 7a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c oz..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61320 req_id = 639f39c3-1f79-462e-bac0-fbe46cb60442
113912025-04-30T06:38:42.119ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28538,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61320 req_id = 639f39c3-1f79-462e-bac0-fbe46cb60442
113922025-04-30T06:38:42.119ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
113932025-04-30T06:38:42.119ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28538, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61320 req_id = 639f39c3-1f79-462e-bac0-fbe46cb60442
113942025-04-30T06:38:42.120ZDEBGtest_quotas (clickhouse-client): unrolling 2 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
113952025-04-30T06:38:42.121ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
113962025-04-30T06:38:42.122ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
113972025-04-30T06:38:42.122ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
113982025-04-30T06:38:42.122ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
113992025-04-30T06:38:42.123ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114002025-04-30T06:38:42.123ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
114012025-04-30T06:38:42.124ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114022025-04-30T06:38:42.125ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
114032025-04-30T06:38:42.125ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
114042025-04-30T06:38:42.125ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "19f0ebf3-c181-4b34-a445-f5ee46881304", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114052025-04-30T06:38:42.126ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
114062025-04-30T06:38:42.143ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = measurements_i64
114072025-04-30T06:38:42.143ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114082025-04-30T06:38:42.145ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60e28541-6010-4e85-ab61-106991d801ba", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114092025-04-30T06:38:42.145ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
114102025-04-30T06:38:42.149ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bc30bd3-e7bb-472a-8ea0-b6cff51582b5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114112025-04-30T06:38:42.149ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
114122025-04-30T06:38:42.153ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e8e6f7c-8934-4a1b-829d-8147bbb7b14b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114132025-04-30T06:38:42.153ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
114142025-04-30T06:38:42.155ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0049a91-ddb1-4b42-b02f-0f3f5623d2f3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114152025-04-30T06:38:42.158ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
114162025-04-30T06:38:42.179ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "620bab21-8fc7-4338-9bc6-942f20d7fdab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114172025-04-30T06:38:42.180ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
114182025-04-30T06:38:42.180ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
114192025-04-30T06:38:42.202ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "112fde7b-78ef-45db-9ab5-33a879bbd968", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114202025-04-30T06:38:42.203ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
114212025-04-30T06:38:42.205ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36a928cc-c46e-4382-b5c1-ed567b8e071b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114222025-04-30T06:38:42.207ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
114232025-04-30T06:38:42.209ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f4df6c7-171f-4ada-9411-a09dd4532abf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114242025-04-30T06:38:42.209ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
114252025-04-30T06:38:42.210ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a52714ab-1e9c-486d-bd74-c1c2bad44508", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114262025-04-30T06:38:42.211ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
114272025-04-30T06:38:42.212ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10befb93-2e00-45bc-871f-e97d1da0049d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114282025-04-30T06:38:42.212ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
114292025-04-30T06:38:42.213ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4109f33b-6788-428e-a506-6f9bb8e02de3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114302025-04-30T06:38:42.213ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
114312025-04-30T06:38:42.230ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87fdc823-2299-46ff-8b1b-d0b89f7a70bb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114322025-04-30T06:38:42.231ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
114332025-04-30T06:38:42.231ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114342025-04-30T06:38:42.231ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114352025-04-30T06:38:42.231ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114362025-04-30T06:38:42.231ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
114372025-04-30T06:38:42.232ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114382025-04-30T06:38:42.232ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
114392025-04-30T06:38:42.235ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114402025-04-30T06:38:42.242ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114412025-04-30T06:38:42.242ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114422025-04-30T06:38:42.242ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
114432025-04-30T06:38:42.242ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114442025-04-30T06:38:42.248ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f06bc9e5-c6ba-4784-970b-bb406816d820", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114452025-04-30T06:38:42.249ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
114462025-04-30T06:38:42.249ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114472025-04-30T06:38:42.249ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
114482025-04-30T06:38:42.250ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114492025-04-30T06:38:42.262ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65e0f478-4964-4b3e-bdb4-7d371f97f95a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114502025-04-30T06:38:42.262ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
114512025-04-30T06:38:42.269ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02d7288d-e47a-4507-a8d1-693747a7d36e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114522025-04-30T06:38:42.269ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
114532025-04-30T06:38:42.271ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5085330-2371-485f-a3aa-b0bae12fc584", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114542025-04-30T06:38:42.271ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
114552025-04-30T06:38:42.274ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4003fbc3-6054-4e70-8e39-4a80e3ed1aa3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114562025-04-30T06:38:42.274ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
114572025-04-30T06:38:42.276ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e026d372-dc92-4805-aa88-94e750c540d6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114582025-04-30T06:38:42.276ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
114592025-04-30T06:38:42.278ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4282d9ed-ec5d-44e6-9f53-c22447bd7bed", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114602025-04-30T06:38:42.278ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
114612025-04-30T06:38:42.281ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f72aabb-a648-4318-a975-cb256bdfc587", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114622025-04-30T06:38:42.281ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
114632025-04-30T06:38:42.283ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4323dfe3-9c02-4a2f-bbb6-7001c020ddb2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114642025-04-30T06:38:42.283ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
114652025-04-30T06:38:42.290ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6611bf72-9dfb-47b0-9078-b5b11050cd86", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114662025-04-30T06:38:42.290ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
114672025-04-30T06:38:42.296ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
114682025-04-30T06:38:42.300ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41f96f63-bad8-4e80-8115-4fcfd5c3b6e4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114692025-04-30T06:38:42.302ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
114702025-04-30T06:38:42.308ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1caa044-3358-4713-b31d-be33c061dd06", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114712025-04-30T06:38:42.308ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
114722025-04-30T06:38:42.310ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e814732-4eac-4c32-82da-870a02983aa7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114732025-04-30T06:38:42.310ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
114742025-04-30T06:38:42.316ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2be7d319-633d-46de-9165-12397580b09f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114752025-04-30T06:38:42.316ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
114762025-04-30T06:38:42.326ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a6defb4-6bf8-404d-929d-a9d36b80ffbe", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114772025-04-30T06:38:42.326ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
114782025-04-30T06:38:42.336ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb8af766-706b-4b50-8d9c-8f426877ea7e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114792025-04-30T06:38:42.336ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
114802025-04-30T06:38:42.338ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37a9a629-9894-4314-a3e7-29d53a3a6ede", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114812025-04-30T06:38:42.338ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
114822025-04-30T06:38:42.339ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da5c9a59-fa16-4fba-8628-882b49674527", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114832025-04-30T06:38:42.339ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
114842025-04-30T06:38:42.340ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4220acd7-643a-42f0-bd3c-bf1ab80e5066", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114852025-04-30T06:38:42.340ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
114862025-04-30T06:38:42.342ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c209c8c7-7359-4ad6-80df-5b986c67d319", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114872025-04-30T06:38:42.342ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
114882025-04-30T06:38:42.343ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09407cbf-048f-4580-adcc-f716cdb53b0a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114892025-04-30T06:38:42.343ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
114902025-04-30T06:38:42.344ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fe4f83e-1c03-48e7-abb2-015c2f4b4d68", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114912025-04-30T06:38:42.345ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
114922025-04-30T06:38:42.346ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c54d235a-303e-40dd-8210-fe57ab136639", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
114932025-04-30T06:38:42.347ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
114942025-04-30T06:38:42.347ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114952025-04-30T06:38:42.347ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114962025-04-30T06:38:42.347ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114972025-04-30T06:38:42.347ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
114982025-04-30T06:38:42.348ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114992025-04-30T06:38:42.358ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115002025-04-30T06:38:42.374ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115012025-04-30T06:38:42.374ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115022025-04-30T06:38:42.374ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
115032025-04-30T06:38:42.374ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115042025-04-30T06:38:42.376ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5194e0e0-0515-4bb2-842f-72a29781a0a7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
115052025-04-30T06:38:42.376ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
115062025-04-30T06:38:42.379ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115072025-04-30T06:38:42.379ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
115082025-04-30T06:38:42.380ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115092025-04-30T06:38:42.384ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14dd3276-2447-441e-9fc7-65e9a4cf2501", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
115102025-04-30T06:38:42.384ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
115112025-04-30T06:38:42.395ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c138796-82d8-41f6-8d6c-64792a5750e5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
115122025-04-30T06:38:42.395ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
115132025-04-30T06:38:42.404ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ced6d55-c438-485a-9b89-a4c6d9c96343", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
115142025-04-30T06:38:42.404ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
115152025-04-30T06:38:42.426ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78760697-76de-4b44-a499-f13fc3d18f6e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
115162025-04-30T06:38:42.426ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
115172025-04-30T06:38:42.446ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
115182025-04-30T06:38:42.446ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f159f162-8580-4f77-92d7-a7409ad47980", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
115192025-04-30T06:38:42.446ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
115202025-04-30T06:38:42.450ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2d6e287-cbbe-4383-bd95-5ade1805f6b0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
115212025-04-30T06:38:42.450ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
115222025-04-30T06:38:42.452ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ef30c47-ce71-4f8a-b761-3c3ed06e743d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
115232025-04-30T06:38:42.456ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
115242025-04-30T06:38:42.462ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6af49666-6e77-4417-8947-eef83fca3ddb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
115252025-04-30T06:38:42.462ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
115262025-04-30T06:38:42.463ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115272025-04-30T06:38:42.463ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115282025-04-30T06:38:42.463ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115292025-04-30T06:38:42.465ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115302025-04-30T06:38:42.469ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115312025-04-30T06:38:42.485ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
115322025-04-30T06:38:42.485ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
115332025-04-30T06:38:42.485ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115342025-04-30T06:38:42.485ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115352025-04-30T06:38:42.485ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115362025-04-30T06:38:42.485ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115372025-04-30T06:38:42.488ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115382025-04-30T06:38:42.494ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
115392025-04-30T06:38:42.494ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115402025-04-30T06:38:42.494ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115412025-04-30T06:38:42.494ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115422025-04-30T06:38:42.502ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115432025-04-30T06:38:42.502ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115442025-04-30T06:38:42.502ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115452025-04-30T06:38:42.502ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
115462025-04-30T06:38:42.503ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115472025-04-30T06:38:42.569ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
115482025-04-30T06:38:42.569ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
115492025-04-30T06:38:42.570ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:44851 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
115502025-04-30T06:38:42.571ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
115512025-04-30T06:38:42.572ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115522025-04-30T06:38:42.588ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115532025-04-30T06:38:42.588ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115542025-04-30T06:38:42.588ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
115552025-04-30T06:38:42.589ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115562025-04-30T06:38:42.589ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:44851 req_id = 3cd5763f-94fd-4d34-8eee-6ca5addfc360 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115572025-04-30T06:38:42.590ZINFOtest_quotas (dropshot): request completed latency_us = 760 local_addr = [::1]:55655 method = GET remote_addr = [::1]:44851 req_id = 3cd5763f-94fd-4d34-8eee-6ca5addfc360 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115582025-04-30T06:38:42.590ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c2 71 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .q..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62198 req_id = 144fb397-ca8e-4e7e-a2d3-7940bf5d164f
115592025-04-30T06:38:42.590ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49777,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62198 req_id = 144fb397-ca8e-4e7e-a2d3-7940bf5d164f
115602025-04-30T06:38:42.591ZDEBGtest_quotas (store): query key key = _dendrite._tcp
115612025-04-30T06:38:42.591ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115622025-04-30T06:38:42.591ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62198 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 144fb397-ca8e-4e7e-a2d3-7940bf5d164f
115632025-04-30T06:38:42.591ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cd5763f-94fd-4d34-8eee-6ca5addfc360", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
115642025-04-30T06:38:42.594ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115652025-04-30T06:38:42.594ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115662025-04-30T06:38:42.595ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
115672025-04-30T06:38:42.597ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115682025-04-30T06:38:42.597ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115692025-04-30T06:38:42.597ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115702025-04-30T06:38:42.598ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115712025-04-30T06:38:42.599ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115722025-04-30T06:38:42.601ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3987, tv_nsec: 400680304 } })
115732025-04-30T06:38:42.601ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
115742025-04-30T06:38:42.602ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115752025-04-30T06:38:42.602ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
115762025-04-30T06:38:42.603ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115772025-04-30T06:38:42.603ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115782025-04-30T06:38:42.608ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
115792025-04-30T06:38:42.608ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115802025-04-30T06:38:42.608ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115812025-04-30T06:38:42.609ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115822025-04-30T06:38:42.612ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 63 41 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e cA..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51532 req_id = 86f963d4-0208-4955-a7a2-5c92de201a23
115832025-04-30T06:38:42.612ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25409,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51532 req_id = 86f963d4-0208-4955-a7a2-5c92de201a23
115842025-04-30T06:38:42.612ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115852025-04-30T06:38:42.612ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51532 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 86f963d4-0208-4955-a7a2-5c92de201a23
115862025-04-30T06:38:42.613ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
115872025-04-30T06:38:42.672ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
115882025-04-30T06:38:42.672ZDEBGtest_quotas (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
115892025-04-30T06:38:42.672ZDEBGtest_quotas (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
115902025-04-30T06:38:42.672ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
115912025-04-30T06:38:42.674ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
115922025-04-30T06:38:42.675ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
115932025-04-30T06:38:42.676ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: dc c1 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:60597 req_id = 23c4363f-06b6-400b-8403-b1bc4141399c
115942025-04-30T06:38:42.676ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56513,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60597 req_id = 23c4363f-06b6-400b-8403-b1bc4141399c
115952025-04-30T06:38:42.677ZDEBGtest_quotas (store): query key key = _dendrite._tcp
115962025-04-30T06:38:42.678ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115972025-04-30T06:38:42.678ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60597 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 23c4363f-06b6-400b-8403-b1bc4141399c
115982025-04-30T06:38:42.681ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3987, tv_nsec: 478059407 } })
115992025-04-30T06:38:42.682ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
116002025-04-30T06:38:42.682ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b3 6c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .l..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61560 req_id = 9b48addd-d1a0-4209-8c90-8d07c8ce4d9d
116012025-04-30T06:38:42.682ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45932,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61560 req_id = 9b48addd-d1a0-4209-8c90-8d07c8ce4d9d
116022025-04-30T06:38:42.684ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116032025-04-30T06:38:42.684ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61560 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9b48addd-d1a0-4209-8c90-8d07c8ce4d9d
116042025-04-30T06:38:42.686ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
116052025-04-30T06:38:42.757ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
116062025-04-30T06:38:42.757ZDEBGtest_quotas (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
116072025-04-30T06:38:42.758ZDEBGtest_quotas (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
116082025-04-30T06:38:42.758ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
116092025-04-30T06:38:42.762ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
116102025-04-30T06:38:42.826ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
116112025-04-30T06:38:42.827ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
116122025-04-30T06:38:42.827ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 827.121376ms iteration = 40 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
116132025-04-30T06:38:42.827ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 41 reason = Signaled
116142025-04-30T06:38:42.827ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
116152025-04-30T06:38:42.827ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116162025-04-30T06:38:42.827ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116172025-04-30T06:38:42.827ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116182025-04-30T06:38:42.828ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116192025-04-30T06:38:42.844ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
116202025-04-30T06:38:42.864ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116212025-04-30T06:38:42.870ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
116222025-04-30T06:38:42.870ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116232025-04-30T06:38:42.870ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116242025-04-30T06:38:42.910ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116252025-04-30T06:38:42.912ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116262025-04-30T06:38:42.912ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116272025-04-30T06:38:42.912ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116282025-04-30T06:38:42.912ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
116292025-04-30T06:38:42.912ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116302025-04-30T06:38:42.913ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116312025-04-30T06:38:42.913ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116322025-04-30T06:38:42.915ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
116332025-04-30T06:38:42.915ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
116342025-04-30T06:38:42.916ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116352025-04-30T06:38:42.917ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116362025-04-30T06:38:42.917ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116372025-04-30T06:38:42.917ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
116382025-04-30T06:38:42.917ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116392025-04-30T06:38:42.926ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116402025-04-30T06:38:42.926ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116412025-04-30T06:38:42.926ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
116422025-04-30T06:38:42.927ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116432025-04-30T06:38:42.927ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 55 1e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 U............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58306 req_id = 29d8649a-1d41-474f-b21f-169ced6033b5
116442025-04-30T06:38:42.928ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21790,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58306 req_id = 29d8649a-1d41-474f-b21f-169ced6033b5
116452025-04-30T06:38:42.928ZDEBGtest_quotas (store): query key key = _mgs._tcp
116462025-04-30T06:38:42.929ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116472025-04-30T06:38:42.929ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58306 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 29d8649a-1d41-474f-b21f-169ced6033b5
116482025-04-30T06:38:42.929ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3987, tv_nsec: 728719901 } })
116492025-04-30T06:38:42.930ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
116502025-04-30T06:38:42.932ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116512025-04-30T06:38:42.932ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
116522025-04-30T06:38:42.933ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116532025-04-30T06:38:42.933ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f8 ff 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54555 req_id = 6307e63e-481d-4bb6-8251-8661e53ff34f
116542025-04-30T06:38:42.934ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63743,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54555 req_id = 6307e63e-481d-4bb6-8251-8661e53ff34f
116552025-04-30T06:38:42.934ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116562025-04-30T06:38:42.934ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54555 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6307e63e-481d-4bb6-8251-8661e53ff34f
116572025-04-30T06:38:42.982ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
116582025-04-30T06:38:42.986ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1b 78 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .x..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61116 req_id = 6f4e4687-ce51-4af0-992e-b7a9fab3bdbb
116592025-04-30T06:38:42.986ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7032,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61116 req_id = 6f4e4687-ce51-4af0-992e-b7a9fab3bdbb
116602025-04-30T06:38:42.987ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
116612025-04-30T06:38:42.987ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7032, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61116 req_id = 6f4e4687-ce51-4af0-992e-b7a9fab3bdbb
116622025-04-30T06:38:42.987ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
116632025-04-30T06:38:42.987ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
116642025-04-30T06:38:42.987ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
116652025-04-30T06:38:43.000ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "beedbc30-5eab-4952-8e77-464a92775401", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
116662025-04-30T06:38:43.001ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
116672025-04-30T06:38:43.012ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35e224a2-65d2-4b5a-9a15-619a6d58fe79", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
116682025-04-30T06:38:43.012ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
116692025-04-30T06:38:43.021ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c773156e-8760-4a7a-ad14-eec8bd76cadb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
116702025-04-30T06:38:43.021ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
116712025-04-30T06:38:43.023ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0fd61dc-08a2-4adb-977e-0b8b55c366ac", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
116722025-04-30T06:38:43.026ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
116732025-04-30T06:38:43.027ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
116742025-04-30T06:38:43.029ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: c8 ea 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:55546 req_id = 60b6aa9e-1113-4281-aca4-3c4980331585
116752025-04-30T06:38:43.029ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51434,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55546 req_id = 60b6aa9e-1113-4281-aca4-3c4980331585
116762025-04-30T06:38:43.029ZDEBGtest_quotas (store): query key key = _dendrite._tcp
116772025-04-30T06:38:43.029ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116782025-04-30T06:38:43.029ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55546 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 60b6aa9e-1113-4281-aca4-3c4980331585
116792025-04-30T06:38:43.030ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3987, tv_nsec: 829469849 } })
116802025-04-30T06:38:43.030ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
116812025-04-30T06:38:43.031ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bf a7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58936 req_id = 44a54054-e62c-4c61-8fad-b76d201ba0c4
116822025-04-30T06:38:43.031ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49063,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58936 req_id = 44a54054-e62c-4c61-8fad-b76d201ba0c4
116832025-04-30T06:38:43.031ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116842025-04-30T06:38:43.031ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58936 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 44a54054-e62c-4c61-8fad-b76d201ba0c4
116852025-04-30T06:38:43.033ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
116862025-04-30T06:38:43.081ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
116872025-04-30T06:38:43.081ZDEBGtest_quotas (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
116882025-04-30T06:38:43.081ZDEBGtest_quotas (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
116892025-04-30T06:38:43.081ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
116902025-04-30T06:38:43.083ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
116912025-04-30T06:38:43.083ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
116922025-04-30T06:38:43.085ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "320cef44-d78b-4f89-a5a8-c2acad95e194", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
116932025-04-30T06:38:43.085ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
116942025-04-30T06:38:43.087ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f92c99ed-f6b8-4df3-a198-20f1f1bb3b51", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
116952025-04-30T06:38:43.087ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
116962025-04-30T06:38:43.089ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "238dc79c-91e9-43c2-98a2-c4aa9e3ed7af", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
116972025-04-30T06:38:43.089ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
116982025-04-30T06:38:43.091ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9d2f4fb-85a9-4df2-8eec-2f67d244091d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
116992025-04-30T06:38:43.091ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
117002025-04-30T06:38:43.092ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8118167-9755-43d7-99fb-70b292c0f1bf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117012025-04-30T06:38:43.092ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
117022025-04-30T06:38:43.094ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46d5eaa5-ea0c-4c53-a272-14bdfcdd3169", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117032025-04-30T06:38:43.094ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
117042025-04-30T06:38:43.095ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4675d814-6cdb-40b5-969d-9ce9565d93c6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117052025-04-30T06:38:43.095ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
117062025-04-30T06:38:43.096ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c357164-cd96-43f3-824c-a68a8b5a300d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117072025-04-30T06:38:43.096ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
117082025-04-30T06:38:43.097ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52a1e482-0f85-4326-9ccd-2bb835d60bd0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117092025-04-30T06:38:43.098ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
117102025-04-30T06:38:43.099ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "120a1837-2ad7-438d-aa7e-5322aa4da7f3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117112025-04-30T06:38:43.099ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
117122025-04-30T06:38:43.101ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "158e027f-4fdd-4e90-801c-8ac5acf07e33", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117132025-04-30T06:38:43.101ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
117142025-04-30T06:38:43.103ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117152025-04-30T06:38:43.103ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117162025-04-30T06:38:43.103ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c2589d1-3cd1-42be-bb0f-01fa2841459a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117172025-04-30T06:38:43.103ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
117182025-04-30T06:38:43.103ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
117192025-04-30T06:38:43.106ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
117202025-04-30T06:38:43.109ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53b3a436-da3e-4b02-851e-fe1ee89e11f7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117212025-04-30T06:38:43.109ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
117222025-04-30T06:38:43.110ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
117232025-04-30T06:38:43.110ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
117242025-04-30T06:38:43.112ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68a0e957-28bc-49b6-9108-047580546c0f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117252025-04-30T06:38:43.112ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
117262025-04-30T06:38:43.113ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
117272025-04-30T06:38:43.113ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117282025-04-30T06:38:43.114ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be335c19-3943-46af-b3d4-b0409852e9a5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117292025-04-30T06:38:43.114ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
117302025-04-30T06:38:43.115ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c84710b-ccff-498e-95ba-a6dabb27131f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117312025-04-30T06:38:43.115ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
117322025-04-30T06:38:43.117ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8e08f7a-7c64-4bfd-bef5-ab2890b2cadb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117332025-04-30T06:38:43.117ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
117342025-04-30T06:38:43.118ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e079b3a2-4293-4d9d-b21d-d9c5aff93665", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117352025-04-30T06:38:43.118ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
117362025-04-30T06:38:43.120ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "784f81a0-ab4f-4777-aefc-8f4faf842c5e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117372025-04-30T06:38:43.120ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
117382025-04-30T06:38:43.121ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be23b270-845a-42be-9a12-aa58f3bf926f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:42 GMT"} })
117392025-04-30T06:38:43.121ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
117402025-04-30T06:38:43.123ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9aba6d85-1ecb-40d4-957f-7d057ee0a11a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117412025-04-30T06:38:43.123ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
117422025-04-30T06:38:43.125ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71109e0f-e8a1-46c0-b18f-f0f8ae8c30bf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117432025-04-30T06:38:43.125ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
117442025-04-30T06:38:43.127ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d20fd69-8662-47cd-96c7-618c01c36beb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117452025-04-30T06:38:43.128ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
117462025-04-30T06:38:43.130ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8b0d84c-7f9d-438a-81f7-8e8bb0046f28", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117472025-04-30T06:38:43.130ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
117482025-04-30T06:38:43.134ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eb85df3-f243-43a9-9da7-45a181d816fd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117492025-04-30T06:38:43.134ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
117502025-04-30T06:38:43.134ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117512025-04-30T06:38:43.134ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117522025-04-30T06:38:43.134ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117532025-04-30T06:38:43.134ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
117542025-04-30T06:38:43.135ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117552025-04-30T06:38:43.138ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b79d467f-b249-4192-9966-81f22ddf5a01", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117562025-04-30T06:38:43.138ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
117572025-04-30T06:38:43.138ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117582025-04-30T06:38:43.144ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117592025-04-30T06:38:43.144ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117602025-04-30T06:38:43.144ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
117612025-04-30T06:38:43.145ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117622025-04-30T06:38:43.147ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c18a20e-d1bd-49c3-af5d-b67585659ad3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117632025-04-30T06:38:43.147ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
117642025-04-30T06:38:43.148ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117652025-04-30T06:38:43.148ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
117662025-04-30T06:38:43.148ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117672025-04-30T06:38:43.149ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf3883fc-7cca-42b0-9256-6d2f6b5fb1dd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117682025-04-30T06:38:43.149ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
117692025-04-30T06:38:43.150ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c5fd1ac-9ead-410a-a5cb-5ad690c345e1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117702025-04-30T06:38:43.150ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
117712025-04-30T06:38:43.152ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bed83f9e-312a-465f-ab8e-18174c7b7614", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117722025-04-30T06:38:43.152ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
117732025-04-30T06:38:43.153ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d30ec75f-5855-42e4-828e-7e59f2e999d6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117742025-04-30T06:38:43.153ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
117752025-04-30T06:38:43.155ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f8305d9-0521-48f3-9555-80abd89fa611", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117762025-04-30T06:38:43.157ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
117772025-04-30T06:38:43.159ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4dae331-f362-4d5b-b6ca-f00f3e7d29c8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117782025-04-30T06:38:43.159ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
117792025-04-30T06:38:43.162ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f045939c-af12-4aae-be43-f245768193c1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117802025-04-30T06:38:43.162ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
117812025-04-30T06:38:43.165ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "052e60c9-62ad-425d-9a76-5c2a5aca3104", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117822025-04-30T06:38:43.166ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
117832025-04-30T06:38:43.168ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ae334c3-44d1-4f74-b7c9-915956a0d7e0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117842025-04-30T06:38:43.168ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
117852025-04-30T06:38:43.171ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfdac99c-974d-40a3-b87f-1c751af9cba2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117862025-04-30T06:38:43.171ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
117872025-04-30T06:38:43.172ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6709e081-c7d2-4eff-b2ca-68f37f7a9b76", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117882025-04-30T06:38:43.172ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
117892025-04-30T06:38:43.173ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae2a6f52-be7c-4864-b1b9-89e25657d922", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117902025-04-30T06:38:43.173ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
117912025-04-30T06:38:43.174ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cd8f7dd-e0cd-4516-ac9f-229fdf2af9bf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117922025-04-30T06:38:43.175ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
117932025-04-30T06:38:43.177ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95d66c71-7860-4041-aa6a-9c60a7fe635c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
117942025-04-30T06:38:43.177ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117952025-04-30T06:38:43.177ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117962025-04-30T06:38:43.177ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117972025-04-30T06:38:43.177ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117982025-04-30T06:38:43.178ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117992025-04-30T06:38:43.178ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
118002025-04-30T06:38:43.181ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118012025-04-30T06:38:43.198ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
118022025-04-30T06:38:43.198ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
118032025-04-30T06:38:43.198ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118042025-04-30T06:38:43.198ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118052025-04-30T06:38:43.198ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118062025-04-30T06:38:43.199ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118072025-04-30T06:38:43.203ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118082025-04-30T06:38:43.211ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
118092025-04-30T06:38:43.211ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118102025-04-30T06:38:43.211ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118112025-04-30T06:38:43.212ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118122025-04-30T06:38:43.213ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
118132025-04-30T06:38:43.275ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
118142025-04-30T06:38:43.276ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
118152025-04-30T06:38:43.276ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118162025-04-30T06:38:43.276ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118172025-04-30T06:38:43.276ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118182025-04-30T06:38:43.276ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
118192025-04-30T06:38:43.277ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118202025-04-30T06:38:43.277ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:56990 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
118212025-04-30T06:38:43.279ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:56990 req_id = 57da2472-fef8-4335-8081-8a6f95ccbc34 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
118222025-04-30T06:38:43.279ZINFOtest_quotas (dropshot): request completed latency_us = 527 local_addr = [::1]:55655 method = GET remote_addr = [::1]:56990 req_id = 57da2472-fef8-4335-8081-8a6f95ccbc34 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
118232025-04-30T06:38:43.279ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "57da2472-fef8-4335-8081-8a6f95ccbc34", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
118242025-04-30T06:38:43.280ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
118252025-04-30T06:38:43.280ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
118262025-04-30T06:38:43.280ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
118272025-04-30T06:38:43.282ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
118282025-04-30T06:38:43.282ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118292025-04-30T06:38:43.282ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118302025-04-30T06:38:43.282ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118312025-04-30T06:38:43.283ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118322025-04-30T06:38:43.283ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118332025-04-30T06:38:43.291ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118342025-04-30T06:38:43.291ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118352025-04-30T06:38:43.291ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
118362025-04-30T06:38:43.292ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118372025-04-30T06:38:43.300ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118382025-04-30T06:38:43.303ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
118392025-04-30T06:38:43.303ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118402025-04-30T06:38:43.303ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118412025-04-30T06:38:43.304ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118422025-04-30T06:38:43.306ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118432025-04-30T06:38:43.306ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
118442025-04-30T06:38:43.307ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118452025-04-30T06:38:43.331ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
118462025-04-30T06:38:43.332ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
118472025-04-30T06:38:43.332ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 505.50698ms iteration = 41 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
118482025-04-30T06:38:43.332ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 42 reason = Signaled
118492025-04-30T06:38:43.332ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
118502025-04-30T06:38:43.332ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118512025-04-30T06:38:43.332ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118522025-04-30T06:38:43.332ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118532025-04-30T06:38:43.334ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118542025-04-30T06:38:43.338ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118552025-04-30T06:38:43.343ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
118562025-04-30T06:38:43.343ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118572025-04-30T06:38:43.343ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118582025-04-30T06:38:43.345ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118592025-04-30T06:38:43.345ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
118602025-04-30T06:38:43.350ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
118612025-04-30T06:38:43.350ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
118622025-04-30T06:38:43.351ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 58 78 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Xx..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57320 req_id = b2d6d81e-852c-4b60-97b2-d8a1468ff101
118632025-04-30T06:38:43.351ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22648,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57320 req_id = b2d6d81e-852c-4b60-97b2-d8a1468ff101
118642025-04-30T06:38:43.351ZDEBGtest_quotas (store): query key key = _mgs._tcp
118652025-04-30T06:38:43.351ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118662025-04-30T06:38:43.351ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57320 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b2d6d81e-852c-4b60-97b2-d8a1468ff101
118672025-04-30T06:38:43.352ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3988, tv_nsec: 151450875 } })
118682025-04-30T06:38:43.352ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
118692025-04-30T06:38:43.353ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 52 13 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e R...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58807 req_id = 06aceb93-9001-4f4d-a660-4c1bc1ac4f36
118702025-04-30T06:38:43.354ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21011,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58807 req_id = 06aceb93-9001-4f4d-a660-4c1bc1ac4f36
118712025-04-30T06:38:43.354ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118722025-04-30T06:38:43.354ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58807 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 06aceb93-9001-4f4d-a660-4c1bc1ac4f36
118732025-04-30T06:38:43.443ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
118742025-04-30T06:38:43.450ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118752025-04-30T06:38:43.450ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118762025-04-30T06:38:43.450ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118772025-04-30T06:38:43.450ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
118782025-04-30T06:38:43.451ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118792025-04-30T06:38:43.452ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 08 83 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65183 req_id = edf538b0-b67e-401f-b268-6974d68d268e
118802025-04-30T06:38:43.452ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2179,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65183 req_id = edf538b0-b67e-401f-b268-6974d68d268e
118812025-04-30T06:38:43.452ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
118822025-04-30T06:38:43.453ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2179, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65183 req_id = edf538b0-b67e-401f-b268-6974d68d268e
118832025-04-30T06:38:43.456ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
118842025-04-30T06:38:43.456ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
118852025-04-30T06:38:43.456ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
118862025-04-30T06:38:43.457ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118872025-04-30T06:38:43.467ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118882025-04-30T06:38:43.467ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118892025-04-30T06:38:43.467ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
118902025-04-30T06:38:43.468ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118912025-04-30T06:38:43.471ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118922025-04-30T06:38:43.471ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
118932025-04-30T06:38:43.472ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118942025-04-30T06:38:43.474ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "072a5543-1261-4afd-9264-2b7684337e42", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
118952025-04-30T06:38:43.474ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
118962025-04-30T06:38:43.477ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d3c67e2-8f23-4232-b9c8-acdc2c76a036", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
118972025-04-30T06:38:43.477ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
118982025-04-30T06:38:43.479ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "24dc6240-3766-4587-b74f-97d1c0646b2a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
118992025-04-30T06:38:43.480ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
119002025-04-30T06:38:43.482ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8b46958-156c-4b63-a5e8-b7efea3e952c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119012025-04-30T06:38:43.482ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
119022025-04-30T06:38:43.485ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1833463d-0658-4420-bf1a-f7ff46c6a4d2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119032025-04-30T06:38:43.485ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
119042025-04-30T06:38:43.488ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "adbb0e81-28ee-4da0-8c5b-527820f2bed4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119052025-04-30T06:38:43.488ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
119062025-04-30T06:38:43.490ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d80a79bf-80ff-4248-9ec9-71bc1c35dbc6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119072025-04-30T06:38:43.490ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
119082025-04-30T06:38:43.492ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec267548-b0f8-41e3-a730-156f6c187be2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119092025-04-30T06:38:43.492ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
119102025-04-30T06:38:43.493ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2eb13685-4eba-4852-b404-fdaecc6054ed", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119112025-04-30T06:38:43.494ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
119122025-04-30T06:38:43.495ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18b3eb36-66a7-4e0a-a62a-2647ab76dd84", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119132025-04-30T06:38:43.495ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
119142025-04-30T06:38:43.496ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6249ba3-f687-4b3c-bcab-7e0a2bf21256", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119152025-04-30T06:38:43.496ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
119162025-04-30T06:38:43.498ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0a30c6b-c330-488c-8e3f-9d70d2fa0316", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119172025-04-30T06:38:43.498ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
119182025-04-30T06:38:43.500ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d31fc6a5-1521-4407-923d-b7ecbb8a24bb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119192025-04-30T06:38:43.500ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
119202025-04-30T06:38:43.505ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
119212025-04-30T06:38:43.505ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1a878f5-9e96-429f-be88-9024048b8f64", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119222025-04-30T06:38:43.505ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
119232025-04-30T06:38:43.509ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "130b70d0-d771-45b1-ad74-aedd0ba24067", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119242025-04-30T06:38:43.509ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
119252025-04-30T06:38:43.511ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e9f07e9-0095-4c45-8bec-85891dd5ff51", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119262025-04-30T06:38:43.511ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
119272025-04-30T06:38:43.513ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcdff19c-1bce-4698-ae61-0c933618c473", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119282025-04-30T06:38:43.514ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
119292025-04-30T06:38:43.516ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1a82d87-96b1-4fc8-9b43-6fea7a5621fc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119302025-04-30T06:38:43.516ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
119312025-04-30T06:38:43.519ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2a7734a-155f-4b9f-9afc-fbb8beece329", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119322025-04-30T06:38:43.519ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
119332025-04-30T06:38:43.520ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97a471f3-b2ff-4607-967e-125a17bc3078", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119342025-04-30T06:38:43.520ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
119352025-04-30T06:38:43.521ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72b2f380-060d-4b90-9cbd-86b85791e0cd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119362025-04-30T06:38:43.521ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
119372025-04-30T06:38:43.522ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7deccafa-f65c-466f-ada7-06755e32eed7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119382025-04-30T06:38:43.522ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
119392025-04-30T06:38:43.523ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b71865b9-aaaa-4846-86a0-bbd7bc48c3d8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119402025-04-30T06:38:43.523ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
119412025-04-30T06:38:43.524ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f21e6254-f7e7-4c4e-ae02-ac1adefbb3de", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119422025-04-30T06:38:43.525ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
119432025-04-30T06:38:43.526ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14ae5f43-2994-4570-a1f4-ca2e2d6be817", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119442025-04-30T06:38:43.527ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
119452025-04-30T06:38:43.529ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a66e2eb7-2242-4831-af6d-9b78d2cc7023", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119462025-04-30T06:38:43.529ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
119472025-04-30T06:38:43.532ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71f8bfe6-cfd9-414c-b33a-7ace72fd7aa6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119482025-04-30T06:38:43.532ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
119492025-04-30T06:38:43.534ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34b4c113-3221-4822-a7ab-3c9a79cf9cf6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119502025-04-30T06:38:43.534ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
119512025-04-30T06:38:43.537ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ead496a-bd9a-4825-baf3-597d9c7f8f5a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119522025-04-30T06:38:43.537ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
119532025-04-30T06:38:43.539ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a827ff48-ecfd-43aa-b5f7-62614516fc1d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119542025-04-30T06:38:43.539ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
119552025-04-30T06:38:43.540ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "479c551f-6b76-47c4-8c6b-cf97f772563e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119562025-04-30T06:38:43.540ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
119572025-04-30T06:38:43.542ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffba1648-982b-4ad7-84a2-6d0031c7e139", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119582025-04-30T06:38:43.542ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
119592025-04-30T06:38:43.543ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83b6da2f-0135-4c9f-bf24-69b4b944d7c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119602025-04-30T06:38:43.543ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
119612025-04-30T06:38:43.544ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08c2c1d3-8576-4fb8-b30b-80ffb2c74b93", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119622025-04-30T06:38:43.544ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
119632025-04-30T06:38:43.546ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ca907ef-53af-4be6-b70b-c57475cfee56", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119642025-04-30T06:38:43.547ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
119652025-04-30T06:38:43.549ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1f29fb3-6149-476f-905a-ea8ac18a4e06", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119662025-04-30T06:38:43.549ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
119672025-04-30T06:38:43.552ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81f4b7ba-0630-429b-9059-d56157940954", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119682025-04-30T06:38:43.554ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
119692025-04-30T06:38:43.556ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119702025-04-30T06:38:43.556ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119712025-04-30T06:38:43.556ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119722025-04-30T06:38:43.556ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
119732025-04-30T06:38:43.556ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119742025-04-30T06:38:43.558ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7eba1a2-634d-411d-a333-a0d90cef84cd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119752025-04-30T06:38:43.558ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
119762025-04-30T06:38:43.559ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119772025-04-30T06:38:43.565ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119782025-04-30T06:38:43.565ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119792025-04-30T06:38:43.565ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
119802025-04-30T06:38:43.565ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119812025-04-30T06:38:43.568ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119822025-04-30T06:38:43.568ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
119832025-04-30T06:38:43.568ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119842025-04-30T06:38:43.572ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0463db0c-5a6a-433c-850b-e4f8ed88801b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119852025-04-30T06:38:43.572ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
119862025-04-30T06:38:43.576ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43cf2f89-5df9-45a5-8fa2-4d8a5ec9ff57", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119872025-04-30T06:38:43.576ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
119882025-04-30T06:38:43.579ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3580d827-9ff1-4148-b53f-320369eabe51", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119892025-04-30T06:38:43.579ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
119902025-04-30T06:38:43.580ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e8f41c3-665f-4f8f-8cdd-f6689bfc2ba4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119912025-04-30T06:38:43.581ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
119922025-04-30T06:38:43.583ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52daf092-01e3-490a-8358-b2af45997962", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119932025-04-30T06:38:43.584ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
119942025-04-30T06:38:43.586ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8ebdce0-7fa0-46e9-8550-1384c19b2174", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119952025-04-30T06:38:43.586ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
119962025-04-30T06:38:43.587ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1c3cd9b-b281-4965-9272-dbd38e50307d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
119972025-04-30T06:38:43.588ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119982025-04-30T06:38:43.588ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119992025-04-30T06:38:43.588ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120002025-04-30T06:38:43.588ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120012025-04-30T06:38:43.589ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120022025-04-30T06:38:43.594ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120032025-04-30T06:38:43.600ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
120042025-04-30T06:38:43.600ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
120052025-04-30T06:38:43.600ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120062025-04-30T06:38:43.600ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120072025-04-30T06:38:43.600ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120082025-04-30T06:38:43.601ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120092025-04-30T06:38:43.603ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120102025-04-30T06:38:43.608ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
120112025-04-30T06:38:43.608ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120122025-04-30T06:38:43.608ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120132025-04-30T06:38:43.608ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120142025-04-30T06:38:43.671ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
120152025-04-30T06:38:43.671ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
120162025-04-30T06:38:43.672ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
120172025-04-30T06:38:43.672ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:38900 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
120182025-04-30T06:38:43.675ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:38900 req_id = 73d3a431-9bc5-4043-85e1-0e1335087efa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
120192025-04-30T06:38:43.675ZINFOtest_quotas (dropshot): request completed latency_us = 533 local_addr = [::1]:55655 method = GET remote_addr = [::1]:38900 req_id = 73d3a431-9bc5-4043-85e1-0e1335087efa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
120202025-04-30T06:38:43.676ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "73d3a431-9bc5-4043-85e1-0e1335087efa", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
120212025-04-30T06:38:43.679ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
120222025-04-30T06:38:43.679ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
120232025-04-30T06:38:43.679ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
120242025-04-30T06:38:43.684ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120252025-04-30T06:38:43.684ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120262025-04-30T06:38:43.684ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120272025-04-30T06:38:43.684ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120282025-04-30T06:38:43.686ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120292025-04-30T06:38:43.689ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120302025-04-30T06:38:43.694ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120312025-04-30T06:38:43.694ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120322025-04-30T06:38:43.694ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120332025-04-30T06:38:43.695ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120342025-04-30T06:38:43.728ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120352025-04-30T06:38:43.728ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120362025-04-30T06:38:43.728ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120372025-04-30T06:38:43.728ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
120382025-04-30T06:38:43.729ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120392025-04-30T06:38:43.731ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120402025-04-30T06:38:43.738ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120412025-04-30T06:38:43.738ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120422025-04-30T06:38:43.738ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
120432025-04-30T06:38:43.739ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120442025-04-30T06:38:43.744ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
120452025-04-30T06:38:43.745ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120462025-04-30T06:38:43.748ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 863f8d3b-9981-4cc2-b719-ab0807ce7aea uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120472025-04-30T06:38:43.748ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 863f8d3b-9981-4cc2-b719-ab0807ce7aea resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120482025-04-30T06:38:43.748ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 863f8d3b-9981-4cc2-b719-ab0807ce7aea roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120492025-04-30T06:38:43.749ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 863f8d3b-9981-4cc2-b719-ab0807ce7aea resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120502025-04-30T06:38:43.756ZINFOtest_quotas (dropshot_internal): request completed latency_us = 8321 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 863f8d3b-9981-4cc2-b719-ab0807ce7aea response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
120512025-04-30T06:38:43.757ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "863f8d3b-9981-4cc2-b719-ab0807ce7aea", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
120522025-04-30T06:38:43.758ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120532025-04-30T06:38:43.758ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
120542025-04-30T06:38:43.758ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
120552025-04-30T06:38:43.758ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 426.05135ms iteration = 42 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
120562025-04-30T06:38:43.758ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 43 reason = Signaled
120572025-04-30T06:38:43.758ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120582025-04-30T06:38:43.758ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120592025-04-30T06:38:43.758ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120602025-04-30T06:38:43.758ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120612025-04-30T06:38:43.759ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120622025-04-30T06:38:43.763ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d9ecb898-7251-4aa3-b2a7-afe8d46cb96c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120632025-04-30T06:38:43.766ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d9ecb898-7251-4aa3-b2a7-afe8d46cb96c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120642025-04-30T06:38:43.766ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d9ecb898-7251-4aa3-b2a7-afe8d46cb96c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120652025-04-30T06:38:43.767ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d9ecb898-7251-4aa3-b2a7-afe8d46cb96c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120662025-04-30T06:38:43.768ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120672025-04-30T06:38:43.772ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120682025-04-30T06:38:43.772ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120692025-04-30T06:38:43.772ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120702025-04-30T06:38:43.772ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120712025-04-30T06:38:43.774ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120722025-04-30T06:38:43.774ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
120732025-04-30T06:38:43.775ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120742025-04-30T06:38:43.777ZINFOtest_quotas (dropshot_internal): request completed latency_us = 14294 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d9ecb898-7251-4aa3-b2a7-afe8d46cb96c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
120752025-04-30T06:38:43.777ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
120762025-04-30T06:38:43.778ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
120772025-04-30T06:38:43.779ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9ecb898-7251-4aa3-b2a7-afe8d46cb96c", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
120782025-04-30T06:38:43.780ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
120792025-04-30T06:38:43.780ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120802025-04-30T06:38:43.780ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
120812025-04-30T06:38:43.781ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c0 69 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .i..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60221 req_id = cefc3a9a-acb3-4c1b-89cc-ff7d92fe691f
120822025-04-30T06:38:43.781ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49257,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60221 req_id = cefc3a9a-acb3-4c1b-89cc-ff7d92fe691f
120832025-04-30T06:38:43.781ZDEBGtest_quotas (store): query key key = _mgs._tcp
120842025-04-30T06:38:43.782ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120852025-04-30T06:38:43.782ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60221 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cefc3a9a-acb3-4c1b-89cc-ff7d92fe691f
120862025-04-30T06:38:43.782ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3988, tv_nsec: 581033871 } })
120872025-04-30T06:38:43.782ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
120882025-04-30T06:38:43.782ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 75 ac 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e u...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57947 req_id = a5953a0b-293d-43f3-a2b6-b8335358462a
120892025-04-30T06:38:43.782ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30124,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57947 req_id = a5953a0b-293d-43f3-a2b6-b8335358462a
120902025-04-30T06:38:43.783ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120912025-04-30T06:38:43.783ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57947 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a5953a0b-293d-43f3-a2b6-b8335358462a
120922025-04-30T06:38:43.859ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
120932025-04-30T06:38:43.860ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120942025-04-30T06:38:43.860ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120952025-04-30T06:38:43.861ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 95 95 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55901 req_id = 73b46ed6-2e6a-4e62-b0f2-f9993da01d6c
120962025-04-30T06:38:43.861ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38293,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55901 req_id = 73b46ed6-2e6a-4e62-b0f2-f9993da01d6c
120972025-04-30T06:38:43.862ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
120982025-04-30T06:38:43.862ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38293, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55901 req_id = 73b46ed6-2e6a-4e62-b0f2-f9993da01d6c
120992025-04-30T06:38:43.862ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
121002025-04-30T06:38:43.863ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
121012025-04-30T06:38:43.863ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
121022025-04-30T06:38:43.863ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121032025-04-30T06:38:43.864ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121042025-04-30T06:38:43.864ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121052025-04-30T06:38:43.864ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121062025-04-30T06:38:43.866ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "538f301e-b1f3-45d9-ab16-b02a75ee015e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121072025-04-30T06:38:43.866ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
121082025-04-30T06:38:43.867ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd37f74e-06ae-4660-a05e-17ebf64a0b0b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121092025-04-30T06:38:43.867ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
121102025-04-30T06:38:43.870ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c62a2754-58bd-46ad-8c04-d1291a640eae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121112025-04-30T06:38:43.870ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
121122025-04-30T06:38:43.872ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07c7b45d-cc63-4d7c-9f12-145d59b7d9ab", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121132025-04-30T06:38:43.873ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
121142025-04-30T06:38:43.876ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61224f7c-bf2f-4f66-95da-bacc42fc1a1e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121152025-04-30T06:38:43.876ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
121162025-04-30T06:38:43.879ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e86e36ee-8477-4371-b5e0-fb571a6cfcf4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121172025-04-30T06:38:43.880ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
121182025-04-30T06:38:43.882ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aadf1578-6443-42d9-8215-bf11addfcf01", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121192025-04-30T06:38:43.882ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
121202025-04-30T06:38:43.888ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d469ac03-3c1c-4e06-bf8b-00b88345f6cc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121212025-04-30T06:38:43.888ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
121222025-04-30T06:38:43.888ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
121232025-04-30T06:38:43.893ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cc58132-fb93-4918-8f62-d536d053168a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121242025-04-30T06:38:43.894ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
121252025-04-30T06:38:43.896ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d885aa0-1357-46a1-9ec5-3e6b712d68d7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121262025-04-30T06:38:43.896ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
121272025-04-30T06:38:43.898ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91acc725-37b7-40d7-b317-5b188957d336", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121282025-04-30T06:38:43.898ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
121292025-04-30T06:38:43.899ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3041ea4f-9507-4f76-a695-e093665c3018", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121302025-04-30T06:38:43.900ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
121312025-04-30T06:38:43.902ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06c2bb7c-c5a9-40be-832b-0590d11eb439", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121322025-04-30T06:38:43.902ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
121332025-04-30T06:38:43.904ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8e857fd-0521-4c44-97f3-fdda6e749d4c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121342025-04-30T06:38:43.905ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
121352025-04-30T06:38:43.908ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d2b8480-b98c-4d67-a5ab-2dcf64fda738", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121362025-04-30T06:38:43.910ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
121372025-04-30T06:38:43.914ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "067c545e-bc32-4581-9682-15eb096b4bd0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121382025-04-30T06:38:43.914ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
121392025-04-30T06:38:43.916ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6ce7e17-cfff-437c-9fbd-e6598038b90b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121402025-04-30T06:38:43.917ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
121412025-04-30T06:38:43.920ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e28a302a-94cf-4087-98cd-94b595de92d6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121422025-04-30T06:38:43.920ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
121432025-04-30T06:38:43.940ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121442025-04-30T06:38:43.941ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121452025-04-30T06:38:43.941ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121462025-04-30T06:38:43.941ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
121472025-04-30T06:38:43.944ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121482025-04-30T06:38:43.953ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121492025-04-30T06:38:43.962ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121502025-04-30T06:38:43.962ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121512025-04-30T06:38:43.962ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
121522025-04-30T06:38:43.962ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121532025-04-30T06:38:43.966ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121542025-04-30T06:38:43.966ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
121552025-04-30T06:38:43.967ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121562025-04-30T06:38:43.967ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "842917d3-a637-4c1a-8fc6-dd15597359c7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121572025-04-30T06:38:43.967ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
121582025-04-30T06:38:43.973ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a39ec02-9a50-422f-9c2b-b5f1be5f7ee6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121592025-04-30T06:38:43.974ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
121602025-04-30T06:38:43.975ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2d1d968-83b2-4ee7-8b26-511fbee5b183", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121612025-04-30T06:38:43.975ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
121622025-04-30T06:38:43.976ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9412bd53-8c88-40a7-87bc-b8de6257d086", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121632025-04-30T06:38:43.976ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
121642025-04-30T06:38:43.977ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f521c55-841b-428e-8148-24e94c0f39e9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121652025-04-30T06:38:43.978ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
121662025-04-30T06:38:43.980ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15fa39c6-972c-4471-92ff-aef48691e255", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121672025-04-30T06:38:43.980ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
121682025-04-30T06:38:43.982ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf34d3d4-7c26-493a-9e06-92a9cd795319", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121692025-04-30T06:38:43.984ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
121702025-04-30T06:38:43.987ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccc24745-3a0f-4f69-8781-cbab1ffb9755", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121712025-04-30T06:38:43.988ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
121722025-04-30T06:38:43.990ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0eae12d-0ac5-4f08-bc97-d5aa6644820e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121732025-04-30T06:38:43.990ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
121742025-04-30T06:38:43.993ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cb45fc3-bef6-4e91-bf45-79b4ce6fcdd6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121752025-04-30T06:38:43.993ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
121762025-04-30T06:38:43.995ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc37bc24-8bf5-4d72-9bfd-f933fe7556a2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121772025-04-30T06:38:43.996ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
121782025-04-30T06:38:43.998ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
121792025-04-30T06:38:44.001ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17a433f9-fcfb-4613-9860-27a9aac66f98", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121802025-04-30T06:38:44.001ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
121812025-04-30T06:38:44.002ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdb19279-e2c7-409f-a3f0-e0ccff67b413", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121822025-04-30T06:38:44.002ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
121832025-04-30T06:38:44.003ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d2387f8-14d8-4a03-9d69-99274ad9a96f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121842025-04-30T06:38:44.003ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
121852025-04-30T06:38:44.004ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b0227ab-1119-4f6b-9b59-6f7dd6943338", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121862025-04-30T06:38:44.004ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
121872025-04-30T06:38:44.006ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c636bb4-bec1-4e0c-8efa-f237dad0811c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121882025-04-30T06:38:44.006ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
121892025-04-30T06:38:44.007ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af4096fa-892d-43f2-828b-91f7a88017e6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121902025-04-30T06:38:44.007ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
121912025-04-30T06:38:44.009ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53f136e7-e734-4615-aa45-f84588a63670", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121922025-04-30T06:38:44.009ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
121932025-04-30T06:38:44.011ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9459151b-65c9-41a5-abae-8fa6a8f4f598", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121942025-04-30T06:38:44.011ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
121952025-04-30T06:38:44.013ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0048faf8-e110-4570-82a8-36686ad51110", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121962025-04-30T06:38:44.014ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
121972025-04-30T06:38:44.016ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f90a1005-3428-4119-9a1a-1e5f5e336f9c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
121982025-04-30T06:38:44.016ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
121992025-04-30T06:38:44.018ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3f31c60-7520-4d0c-bf2c-3d0d32527cfa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
122002025-04-30T06:38:44.018ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
122012025-04-30T06:38:44.020ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "187a9b75-86fa-482e-a47b-ca1621315b2d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
122022025-04-30T06:38:44.020ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
122032025-04-30T06:38:44.021ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c98f159-f3be-4698-8d34-30b29db3538f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
122042025-04-30T06:38:44.021ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
122052025-04-30T06:38:44.023ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5861382d-ccb6-4cfe-8245-af551b0ec1fc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
122062025-04-30T06:38:44.023ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
122072025-04-30T06:38:44.028ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a533fb0c-2b84-4be2-a0a5-b13e6fdcb488", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
122082025-04-30T06:38:44.028ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
122092025-04-30T06:38:44.038ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53175981-6f53-4c96-b497-5bc5774d0274", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:43 GMT"} })
122102025-04-30T06:38:44.038ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122112025-04-30T06:38:44.038ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122122025-04-30T06:38:44.038ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122132025-04-30T06:38:44.038ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122142025-04-30T06:38:44.039ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122152025-04-30T06:38:44.051ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122162025-04-30T06:38:44.051ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122172025-04-30T06:38:44.051ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122182025-04-30T06:38:44.051ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
122192025-04-30T06:38:44.052ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122202025-04-30T06:38:44.052ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122212025-04-30T06:38:44.056ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122222025-04-30T06:38:44.056ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122232025-04-30T06:38:44.056ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122242025-04-30T06:38:44.056ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122252025-04-30T06:38:44.056ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122262025-04-30T06:38:44.057ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122272025-04-30T06:38:44.059ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122282025-04-30T06:38:44.065ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122292025-04-30T06:38:44.065ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122302025-04-30T06:38:44.065ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
122312025-04-30T06:38:44.066ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122322025-04-30T06:38:44.067ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122332025-04-30T06:38:44.073ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122342025-04-30T06:38:44.073ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122352025-04-30T06:38:44.073ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122362025-04-30T06:38:44.074ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122372025-04-30T06:38:44.074ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122382025-04-30T06:38:44.074ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
122392025-04-30T06:38:44.074ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122402025-04-30T06:38:44.138ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
122412025-04-30T06:38:44.138ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
122422025-04-30T06:38:44.147ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:57442 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
122432025-04-30T06:38:44.147ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122442025-04-30T06:38:44.147ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122452025-04-30T06:38:44.148ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
122462025-04-30T06:38:44.149ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
122472025-04-30T06:38:44.152ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:57442 req_id = 38ae1419-59d7-4bab-a74d-1d361f7ecbca sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122482025-04-30T06:38:44.156ZINFOtest_quotas (dropshot): request completed latency_us = 3959 local_addr = [::1]:55655 method = GET remote_addr = [::1]:57442 req_id = 38ae1419-59d7-4bab-a74d-1d361f7ecbca response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122492025-04-30T06:38:44.158ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
122502025-04-30T06:38:44.158ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
122512025-04-30T06:38:44.158ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "38ae1419-59d7-4bab-a74d-1d361f7ecbca", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
122522025-04-30T06:38:44.160ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122532025-04-30T06:38:44.160ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122542025-04-30T06:38:44.160ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
122552025-04-30T06:38:44.163ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122562025-04-30T06:38:44.163ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122572025-04-30T06:38:44.163ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122582025-04-30T06:38:44.163ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122592025-04-30T06:38:44.164ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122602025-04-30T06:38:44.169ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
122612025-04-30T06:38:44.169ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122622025-04-30T06:38:44.170ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122632025-04-30T06:38:44.175ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122642025-04-30T06:38:44.175ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122652025-04-30T06:38:44.175ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122662025-04-30T06:38:44.176ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122672025-04-30T06:38:44.210ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
122682025-04-30T06:38:44.221ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
122692025-04-30T06:38:44.221ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
122702025-04-30T06:38:44.221ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 462.698206ms iteration = 43 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
122712025-04-30T06:38:44.221ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 44 reason = Signaled
122722025-04-30T06:38:44.221ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122732025-04-30T06:38:44.221ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122742025-04-30T06:38:44.221ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122752025-04-30T06:38:44.221ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122762025-04-30T06:38:44.222ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122772025-04-30T06:38:44.225ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122782025-04-30T06:38:44.229ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122792025-04-30T06:38:44.230ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122802025-04-30T06:38:44.230ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122812025-04-30T06:38:44.230ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122822025-04-30T06:38:44.233ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
122832025-04-30T06:38:44.233ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
122842025-04-30T06:38:44.235ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0c fa 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60756 req_id = 75b6efc2-b5f6-4788-9b25-195f183204e0
122852025-04-30T06:38:44.237ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3322,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60756 req_id = 75b6efc2-b5f6-4788-9b25-195f183204e0
122862025-04-30T06:38:44.237ZDEBGtest_quotas (store): query key key = _mgs._tcp
122872025-04-30T06:38:44.237ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122882025-04-30T06:38:44.237ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60756 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 75b6efc2-b5f6-4788-9b25-195f183204e0
122892025-04-30T06:38:44.238ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3989, tv_nsec: 37491062 } })
122902025-04-30T06:38:44.238ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
122912025-04-30T06:38:44.238ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1d 8c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62028 req_id = 9e4dc430-54db-4399-aaa6-459ca4692a55
122922025-04-30T06:38:44.239ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7564,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62028 req_id = 9e4dc430-54db-4399-aaa6-459ca4692a55
122932025-04-30T06:38:44.239ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122942025-04-30T06:38:44.239ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62028 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9e4dc430-54db-4399-aaa6-459ca4692a55
122952025-04-30T06:38:44.315ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
122962025-04-30T06:38:44.315ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122972025-04-30T06:38:44.315ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122982025-04-30T06:38:44.315ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122992025-04-30T06:38:44.316ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
123002025-04-30T06:38:44.316ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123012025-04-30T06:38:44.316ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
123022025-04-30T06:38:44.317ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 75 55 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c uU..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58881 req_id = e9177fe7-21aa-479f-8cb5-501c1759896e
123032025-04-30T06:38:44.317ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30037,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58881 req_id = e9177fe7-21aa-479f-8cb5-501c1759896e
123042025-04-30T06:38:44.317ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
123052025-04-30T06:38:44.317ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30037, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58881 req_id = e9177fe7-21aa-479f-8cb5-501c1759896e
123062025-04-30T06:38:44.318ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
123072025-04-30T06:38:44.318ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
123082025-04-30T06:38:44.318ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
123092025-04-30T06:38:44.320ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123102025-04-30T06:38:44.327ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123112025-04-30T06:38:44.327ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123122025-04-30T06:38:44.327ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
123132025-04-30T06:38:44.328ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123142025-04-30T06:38:44.330ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "60c907c8-a5af-41ad-b7bb-564e97ead431", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123152025-04-30T06:38:44.331ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
123162025-04-30T06:38:44.332ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123172025-04-30T06:38:44.332ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
123182025-04-30T06:38:44.335ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123192025-04-30T06:38:44.336ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f06cdb9c-4c36-4f13-8e8f-b37d477fb6a1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123202025-04-30T06:38:44.336ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
123212025-04-30T06:38:44.339ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49ebb9ea-4803-4ac1-bd70-9fc244cef2cd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123222025-04-30T06:38:44.339ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
123232025-04-30T06:38:44.342ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "010a400d-6999-4ab8-b164-6985fc0b4fe0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123242025-04-30T06:38:44.342ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
123252025-04-30T06:38:44.346ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "482378a8-e8ef-4071-b8fb-cefad446a9d0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123262025-04-30T06:38:44.346ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
123272025-04-30T06:38:44.350ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f28a2160-9434-4220-a53e-2c882f79185b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123282025-04-30T06:38:44.351ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
123292025-04-30T06:38:44.357ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbd58146-7adc-4130-bf6d-f74a570bffb4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123302025-04-30T06:38:44.358ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
123312025-04-30T06:38:44.362ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "caa4ea8e-e14d-4943-a7ba-0df3e23bfc62", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123322025-04-30T06:38:44.363ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
123332025-04-30T06:38:44.365ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2299d96a-bceb-4dfd-b29f-d5ea3acfce0c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123342025-04-30T06:38:44.365ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
123352025-04-30T06:38:44.366ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
123362025-04-30T06:38:44.367ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10584773-cd22-4fda-9e1e-986dcc096eab", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123372025-04-30T06:38:44.368ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
123382025-04-30T06:38:44.369ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75d6a705-704b-41a3-bbd0-a1879b96baef", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123392025-04-30T06:38:44.369ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
123402025-04-30T06:38:44.370ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef9c66a3-57eb-4943-b15f-562fa4911058", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123412025-04-30T06:38:44.370ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
123422025-04-30T06:38:44.372ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d261ce40-0fc1-4dce-98bb-8cb1969f2329", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123432025-04-30T06:38:44.372ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
123442025-04-30T06:38:44.376ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a835f668-dde4-402d-82bc-13bf30da63ad", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123452025-04-30T06:38:44.377ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
123462025-04-30T06:38:44.379ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e342e15f-fb0a-4ca8-a94a-e19e7b2fbe2e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123472025-04-30T06:38:44.380ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
123482025-04-30T06:38:44.382ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3ae6d96-72a0-46b4-a3ae-b078ea12ea12", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123492025-04-30T06:38:44.382ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
123502025-04-30T06:38:44.384ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "655a0f11-05e3-4c61-8e18-c92d37127e65", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123512025-04-30T06:38:44.384ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
123522025-04-30T06:38:44.386ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1ce5a84-dd6e-48e5-ac70-0f55b3d1b61d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123532025-04-30T06:38:44.387ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
123542025-04-30T06:38:44.389ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b40b64e9-de0c-4907-9027-eb0f1909d2bc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123552025-04-30T06:38:44.389ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
123562025-04-30T06:38:44.390ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc6f0bb6-2d18-43d7-bfd4-3528bf413747", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123572025-04-30T06:38:44.391ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
123582025-04-30T06:38:44.392ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d75827f-a044-4549-99a9-b635c61df7b8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123592025-04-30T06:38:44.392ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
123602025-04-30T06:38:44.393ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34bab520-b18a-4a2c-b0e2-110044c7e365", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123612025-04-30T06:38:44.393ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
123622025-04-30T06:38:44.394ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "beda9694-d806-4136-a8af-72b99a63a956", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123632025-04-30T06:38:44.394ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
123642025-04-30T06:38:44.395ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7566496-2f5b-4ec0-955a-99adb35f2df9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123652025-04-30T06:38:44.395ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
123662025-04-30T06:38:44.397ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d3d18fe-bddc-43a8-b59e-906222ee40b5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123672025-04-30T06:38:44.397ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
123682025-04-30T06:38:44.400ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd7aaed6-4b47-4527-971c-6cf3696a53f5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123692025-04-30T06:38:44.400ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
123702025-04-30T06:38:44.402ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f14e629-b763-4454-b2b8-2b42667777a9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123712025-04-30T06:38:44.402ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
123722025-04-30T06:38:44.406ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8da688c6-de17-4c3d-bcec-cae669fa58e2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123732025-04-30T06:38:44.406ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
123742025-04-30T06:38:44.408ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ce861da-d1a1-4e88-8bb8-c923741811f7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123752025-04-30T06:38:44.408ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
123762025-04-30T06:38:44.410ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb0b8acd-801a-4b42-8c96-ea6a66426e64", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123772025-04-30T06:38:44.410ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
123782025-04-30T06:38:44.411ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63d431b1-c6f7-4a45-9cdf-1ddc22e18be2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123792025-04-30T06:38:44.411ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
123802025-04-30T06:38:44.412ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f354f354-bd0f-4f20-996d-131a56416598", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123812025-04-30T06:38:44.412ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
123822025-04-30T06:38:44.413ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32071727-20fb-40b3-86d9-336a28d62e13", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123832025-04-30T06:38:44.413ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
123842025-04-30T06:38:44.415ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fa5ee4a-0aa8-490e-9fef-18de586168ee", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123852025-04-30T06:38:44.415ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
123862025-04-30T06:38:44.416ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2136cb82-16bf-4fa7-8058-229b0fbbfb61", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123872025-04-30T06:38:44.416ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
123882025-04-30T06:38:44.417ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123892025-04-30T06:38:44.417ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123902025-04-30T06:38:44.417ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123912025-04-30T06:38:44.417ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
123922025-04-30T06:38:44.419ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123932025-04-30T06:38:44.424ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de9d3efe-3db1-4f31-ab23-b5be350c68f5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
123942025-04-30T06:38:44.425ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
123952025-04-30T06:38:44.430ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123962025-04-30T06:38:44.441ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123972025-04-30T06:38:44.441ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123982025-04-30T06:38:44.441ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
123992025-04-30T06:38:44.442ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124002025-04-30T06:38:44.444ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49856b29-2c33-4f11-9014-4cfbb20000a0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
124012025-04-30T06:38:44.444ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
124022025-04-30T06:38:44.445ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124032025-04-30T06:38:44.445ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
124042025-04-30T06:38:44.446ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124052025-04-30T06:38:44.455ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e877d1f-ce1d-4b94-90c0-ab0463f94beb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
124062025-04-30T06:38:44.456ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
124072025-04-30T06:38:44.476ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dcde6ce-9e29-4fa4-a583-9fc7c6a70f07", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
124082025-04-30T06:38:44.477ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
124092025-04-30T06:38:44.479ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9efdee45-ad2f-401c-bafe-0f2afc6745c0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
124102025-04-30T06:38:44.479ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
124112025-04-30T06:38:44.481ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cdf6b53-ddd0-436b-aca2-dce21209131c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
124122025-04-30T06:38:44.481ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
124132025-04-30T06:38:44.482ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e92974bb-a1f7-411d-a676-68dac0d6a731", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
124142025-04-30T06:38:44.482ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
124152025-04-30T06:38:44.484ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0daa0815-0845-436a-9359-d99d518357a3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
124162025-04-30T06:38:44.484ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
124172025-04-30T06:38:44.486ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bd77f42-c183-4a87-9a1f-ed231d13f76a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
124182025-04-30T06:38:44.486ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
124192025-04-30T06:38:44.487ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e1aae05-a9f8-4caf-8afd-e978605e31ac", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
124202025-04-30T06:38:44.487ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124212025-04-30T06:38:44.487ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124222025-04-30T06:38:44.487ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124232025-04-30T06:38:44.487ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124242025-04-30T06:38:44.488ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124252025-04-30T06:38:44.493ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124262025-04-30T06:38:44.498ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124272025-04-30T06:38:44.498ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
124282025-04-30T06:38:44.498ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124292025-04-30T06:38:44.498ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124302025-04-30T06:38:44.498ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124312025-04-30T06:38:44.499ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124322025-04-30T06:38:44.503ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
124332025-04-30T06:38:44.516ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124342025-04-30T06:38:44.520ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
124352025-04-30T06:38:44.520ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124362025-04-30T06:38:44.520ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124372025-04-30T06:38:44.521ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124382025-04-30T06:38:44.644ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
124392025-04-30T06:38:44.644ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
124402025-04-30T06:38:44.645ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124412025-04-30T06:38:44.645ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124422025-04-30T06:38:44.645ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124432025-04-30T06:38:44.645ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
124442025-04-30T06:38:44.646ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124452025-04-30T06:38:44.646ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:52681 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
124462025-04-30T06:38:44.646ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:52681 req_id = 51072966-834b-434b-a2cf-89559556f8f8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
124472025-04-30T06:38:44.647ZINFOtest_quotas (dropshot): request completed latency_us = 620 local_addr = [::1]:55655 method = GET remote_addr = [::1]:52681 req_id = 51072966-834b-434b-a2cf-89559556f8f8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
124482025-04-30T06:38:44.648ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "51072966-834b-434b-a2cf-89559556f8f8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
124492025-04-30T06:38:44.651ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124502025-04-30T06:38:44.651ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124512025-04-30T06:38:44.651ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
124522025-04-30T06:38:44.654ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124532025-04-30T06:38:44.654ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124542025-04-30T06:38:44.654ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124552025-04-30T06:38:44.654ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124562025-04-30T06:38:44.655ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124572025-04-30T06:38:44.655ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124582025-04-30T06:38:44.664ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124592025-04-30T06:38:44.665ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124602025-04-30T06:38:44.665ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
124612025-04-30T06:38:44.667ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124622025-04-30T06:38:44.669ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124632025-04-30T06:38:44.673ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
124642025-04-30T06:38:44.674ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124652025-04-30T06:38:44.674ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124662025-04-30T06:38:44.674ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124672025-04-30T06:38:44.675ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124682025-04-30T06:38:44.675ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
124692025-04-30T06:38:44.676ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124702025-04-30T06:38:44.676ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
124712025-04-30T06:38:44.678ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 90 2f 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ./..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61327 req_id = 2c2a1ce3-2430-483a-b8c9-d870d3a4437b
124722025-04-30T06:38:44.679ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36911,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61327 req_id = 2c2a1ce3-2430-483a-b8c9-d870d3a4437b
124732025-04-30T06:38:44.679ZDEBGtest_quotas (store): query key key = _dendrite._tcp
124742025-04-30T06:38:44.679ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124752025-04-30T06:38:44.679ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61327 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2c2a1ce3-2430-483a-b8c9-d870d3a4437b
124762025-04-30T06:38:44.680ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3989, tv_nsec: 478945241 } })
124772025-04-30T06:38:44.680ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
124782025-04-30T06:38:44.680ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 53 a5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e S...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60643 req_id = 17fd8add-9ba2-420c-bc36-73170c230e2b
124792025-04-30T06:38:44.680ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21413,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60643 req_id = 17fd8add-9ba2-420c-bc36-73170c230e2b
124802025-04-30T06:38:44.680ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124812025-04-30T06:38:44.680ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60643 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 17fd8add-9ba2-420c-bc36-73170c230e2b
124822025-04-30T06:38:44.681ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
124832025-04-30T06:38:44.771ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
124842025-04-30T06:38:44.771ZDEBGtest_quotas (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
124852025-04-30T06:38:44.771ZDEBGtest_quotas (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
124862025-04-30T06:38:44.771ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
124872025-04-30T06:38:44.773ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
124882025-04-30T06:38:44.775ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
124892025-04-30T06:38:44.776ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: df 60 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .`..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52832 req_id = 6feda60d-7fd8-47aa-90e8-2829e7094984
124902025-04-30T06:38:44.776ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57184,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52832 req_id = 6feda60d-7fd8-47aa-90e8-2829e7094984
124912025-04-30T06:38:44.776ZDEBGtest_quotas (store): query key key = _dendrite._tcp
124922025-04-30T06:38:44.776ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124932025-04-30T06:38:44.776ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52832 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6feda60d-7fd8-47aa-90e8-2829e7094984
124942025-04-30T06:38:44.777ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3989, tv_nsec: 576194153 } })
124952025-04-30T06:38:44.777ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
124962025-04-30T06:38:44.777ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b2 f9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53709 req_id = fd77e1c9-6772-4663-a688-a4c7199a82b8
124972025-04-30T06:38:44.777ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45817,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53709 req_id = fd77e1c9-6772-4663-a688-a4c7199a82b8
124982025-04-30T06:38:44.777ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124992025-04-30T06:38:44.777ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53709 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fd77e1c9-6772-4663-a688-a4c7199a82b8
125002025-04-30T06:38:44.778ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
125012025-04-30T06:38:44.830ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
125022025-04-30T06:38:44.830ZDEBGtest_quotas (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
125032025-04-30T06:38:44.830ZDEBGtest_quotas (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
125042025-04-30T06:38:44.830ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
125052025-04-30T06:38:44.830ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
125062025-04-30T06:38:44.857ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
125072025-04-30T06:38:44.857ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
125082025-04-30T06:38:44.864ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
125092025-04-30T06:38:44.865ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
125102025-04-30T06:38:44.865ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
125112025-04-30T06:38:44.865ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
125122025-04-30T06:38:44.875ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
125132025-04-30T06:38:44.876ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
125142025-04-30T06:38:44.876ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 654.471923ms iteration = 44 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
125152025-04-30T06:38:44.876ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 45 reason = Signaled
125162025-04-30T06:38:44.876ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
125172025-04-30T06:38:44.876ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125182025-04-30T06:38:44.876ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125192025-04-30T06:38:44.876ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125202025-04-30T06:38:44.877ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125212025-04-30T06:38:44.878ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
125222025-04-30T06:38:44.880ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125232025-04-30T06:38:44.884ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
125242025-04-30T06:38:44.884ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125252025-04-30T06:38:44.884ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125262025-04-30T06:38:44.885ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125272025-04-30T06:38:44.889ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
125282025-04-30T06:38:44.889ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
125292025-04-30T06:38:44.890ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 03 d9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60608 req_id = d0191302-2a05-459b-9a89-58a2767c8c48
125302025-04-30T06:38:44.890ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 985,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60608 req_id = d0191302-2a05-459b-9a89-58a2767c8c48
125312025-04-30T06:38:44.890ZDEBGtest_quotas (store): query key key = _mgs._tcp
125322025-04-30T06:38:44.891ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125332025-04-30T06:38:44.891ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60608 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d0191302-2a05-459b-9a89-58a2767c8c48
125342025-04-30T06:38:44.892ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3989, tv_nsec: 691194543 } })
125352025-04-30T06:38:44.892ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
125362025-04-30T06:38:44.892ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 09 47 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .G..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60232 req_id = c2bea69a-d20c-4d6f-bbf3-113f45c92b6a
125372025-04-30T06:38:44.892ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2375,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60232 req_id = c2bea69a-d20c-4d6f-bbf3-113f45c92b6a
125382025-04-30T06:38:44.892ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125392025-04-30T06:38:44.893ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60232 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c2bea69a-d20c-4d6f-bbf3-113f45c92b6a
125402025-04-30T06:38:44.960ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
125412025-04-30T06:38:44.962ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125422025-04-30T06:38:44.962ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125432025-04-30T06:38:44.962ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125442025-04-30T06:38:44.962ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
125452025-04-30T06:38:44.963ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125462025-04-30T06:38:44.965ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9a c0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63172 req_id = b9281e65-9d6c-42d3-a7b5-9c45601c69d4
125472025-04-30T06:38:44.965ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39616,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63172 req_id = b9281e65-9d6c-42d3-a7b5-9c45601c69d4
125482025-04-30T06:38:44.965ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
125492025-04-30T06:38:44.966ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39616, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63172 req_id = b9281e65-9d6c-42d3-a7b5-9c45601c69d4
125502025-04-30T06:38:44.969ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
125512025-04-30T06:38:44.969ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
125522025-04-30T06:38:44.969ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
125532025-04-30T06:38:44.969ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125542025-04-30T06:38:44.981ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125552025-04-30T06:38:44.981ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125562025-04-30T06:38:44.981ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
125572025-04-30T06:38:44.982ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125582025-04-30T06:38:44.989ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125592025-04-30T06:38:44.989ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
125602025-04-30T06:38:44.990ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125612025-04-30T06:38:45.006ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "30201b7f-c6ea-4c65-a5ad-ec92f5cfe4a3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125622025-04-30T06:38:45.006ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
125632025-04-30T06:38:45.019ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dc350b2-4c3e-4882-a253-aae0513e0b51", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125642025-04-30T06:38:45.019ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
125652025-04-30T06:38:45.028ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bab1216a-f5ea-44ec-939d-85a80c7732a7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125662025-04-30T06:38:45.029ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
125672025-04-30T06:38:45.031ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "546a5132-4a12-4197-9f7d-1018c5ee33e3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125682025-04-30T06:38:45.032ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
125692025-04-30T06:38:45.034ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62b6cdcf-0859-4224-b010-795b73ad0eb0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125702025-04-30T06:38:45.034ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
125712025-04-30T06:38:45.036ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ee50191-7054-474b-9870-57da91c692ab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125722025-04-30T06:38:45.036ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
125732025-04-30T06:38:45.039ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4111c508-84ac-422a-bccb-246c4dfbb197", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125742025-04-30T06:38:45.039ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
125752025-04-30T06:38:45.041ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "816681c7-d9b9-4e92-9c6b-1e5ec8f3f4eb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125762025-04-30T06:38:45.041ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
125772025-04-30T06:38:45.044ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
125782025-04-30T06:38:45.045ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c996f6ef-e215-4035-aaf2-5f638fc101c6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125792025-04-30T06:38:45.045ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
125802025-04-30T06:38:45.046ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f0a4f16-b242-4f25-b1bf-6afbb7dd688d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125812025-04-30T06:38:45.046ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
125822025-04-30T06:38:45.048ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3462cf95-f4c1-43f0-a944-269b0c407db7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125832025-04-30T06:38:45.048ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
125842025-04-30T06:38:45.049ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8947f8b9-6de2-444f-b079-eecf44244d6c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125852025-04-30T06:38:45.049ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
125862025-04-30T06:38:45.050ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5aa08c3-caba-4950-85d9-0b81715f8eb7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125872025-04-30T06:38:45.053ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
125882025-04-30T06:38:45.055ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9355695-9aac-4de7-9e1f-2011804f3d95", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125892025-04-30T06:38:45.055ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
125902025-04-30T06:38:45.057ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9182ef9f-ba3a-40ca-a661-03dbf3a9149a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125912025-04-30T06:38:45.057ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
125922025-04-30T06:38:45.060ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e63886d2-f14d-4a9c-8852-4cd45e970232", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125932025-04-30T06:38:45.060ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
125942025-04-30T06:38:45.064ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aaa9478-5013-4549-82a8-99fa5f40c945", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125952025-04-30T06:38:45.064ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
125962025-04-30T06:38:45.066ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de429e76-dec0-448e-b811-62bb5d8deeda", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125972025-04-30T06:38:45.067ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
125982025-04-30T06:38:45.069ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d365bee-7939-4849-9b84-5efca9625482", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
125992025-04-30T06:38:45.069ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
126002025-04-30T06:38:45.070ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "582bbd1e-f536-42e8-9424-c0571e3e58ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
126012025-04-30T06:38:45.070ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
126022025-04-30T06:38:45.072ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae614efd-7f42-4633-ba9d-18d2b629e861", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
126032025-04-30T06:38:45.073ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
126042025-04-30T06:38:45.074ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd3d86b-a873-41b7-a92b-5179a1d44690", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
126052025-04-30T06:38:45.074ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
126062025-04-30T06:38:45.075ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bd47b4c-46c5-40b3-a9aa-67eeef180688", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
126072025-04-30T06:38:45.078ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
126082025-04-30T06:38:45.080ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a13a9a90-57a7-401f-8bdf-bdbfe735d67d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
126092025-04-30T06:38:45.080ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
126102025-04-30T06:38:45.083ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86425f0c-4af7-4591-835a-107c844ee40d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:44 GMT"} })
126112025-04-30T06:38:45.084ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
126122025-04-30T06:38:45.084ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
126132025-04-30T06:38:45.085ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: e7 1a 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:58940 req_id = b8588917-efdb-4f24-bc08-01b188c26bef
126142025-04-30T06:38:45.085ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59162,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58940 req_id = b8588917-efdb-4f24-bc08-01b188c26bef
126152025-04-30T06:38:45.086ZDEBGtest_quotas (store): query key key = _dendrite._tcp
126162025-04-30T06:38:45.087ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126172025-04-30T06:38:45.087ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58940 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b8588917-efdb-4f24-bc08-01b188c26bef
126182025-04-30T06:38:45.088ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3989, tv_nsec: 887043660 } })
126192025-04-30T06:38:45.088ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
126202025-04-30T06:38:45.088ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 32 44 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 2D..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51267 req_id = e1d780d7-ea6e-4df9-969f-273640f996e4
126212025-04-30T06:38:45.088ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12868,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51267 req_id = e1d780d7-ea6e-4df9-969f-273640f996e4
126222025-04-30T06:38:45.088ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126232025-04-30T06:38:45.088ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51267 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e1d780d7-ea6e-4df9-969f-273640f996e4
126242025-04-30T06:38:45.089ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
126252025-04-30T06:38:45.184ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
126262025-04-30T06:38:45.185ZDEBGtest_quotas (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
126272025-04-30T06:38:45.188ZDEBGtest_quotas (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
126282025-04-30T06:38:45.188ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
126292025-04-30T06:38:45.193ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
126302025-04-30T06:38:45.193ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126312025-04-30T06:38:45.193ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126322025-04-30T06:38:45.193ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126332025-04-30T06:38:45.194ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
126342025-04-30T06:38:45.195ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126352025-04-30T06:38:45.195ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
126362025-04-30T06:38:45.195ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
126372025-04-30T06:38:45.196ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
126382025-04-30T06:38:45.198ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
126392025-04-30T06:38:45.200ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "406df52b-e79c-46a1-9661-54b6aa047cac", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126402025-04-30T06:38:45.200ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
126412025-04-30T06:38:45.200ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126422025-04-30T06:38:45.215ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126432025-04-30T06:38:45.215ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126442025-04-30T06:38:45.215ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
126452025-04-30T06:38:45.216ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126462025-04-30T06:38:45.220ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126472025-04-30T06:38:45.220ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
126482025-04-30T06:38:45.223ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126492025-04-30T06:38:45.231ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
126502025-04-30T06:38:45.231ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
126512025-04-30T06:38:45.231ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "728511f3-bed0-4b23-af79-6de220266cc1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126522025-04-30T06:38:45.231ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
126532025-04-30T06:38:45.245ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3986d29b-e8f9-4043-b852-1b939427451c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126542025-04-30T06:38:45.245ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
126552025-04-30T06:38:45.248ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "173bf9ca-ee52-426a-9ac6-d0578f60386e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126562025-04-30T06:38:45.248ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
126572025-04-30T06:38:45.251ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
126582025-04-30T06:38:45.251ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
126592025-04-30T06:38:45.257ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a3d96bb-27f9-4bb8-8859-67c7609a1f25", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126602025-04-30T06:38:45.257ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
126612025-04-30T06:38:45.259ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c88fc394-b06c-40b6-8b7e-f4b0b53ba59e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126622025-04-30T06:38:45.259ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
126632025-04-30T06:38:45.288ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87815f3c-45d9-4a86-891b-96e07ed2ee30", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126642025-04-30T06:38:45.289ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
126652025-04-30T06:38:45.290ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e37e5f3f-ee47-4d86-ac8e-cf275b6f6eaf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126662025-04-30T06:38:45.290ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
126672025-04-30T06:38:45.293ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04d32ef2-9bf7-4c97-86bc-99953a37d7bb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126682025-04-30T06:38:45.293ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
126692025-04-30T06:38:45.295ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77aea828-258e-43e6-86a7-10b603bc2ca5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126702025-04-30T06:38:45.295ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
126712025-04-30T06:38:45.297ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
126722025-04-30T06:38:45.298ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6dafad4-fe7a-45f6-a671-4908675e6f92", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126732025-04-30T06:38:45.298ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
126742025-04-30T06:38:45.301ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6779037d-e762-4a40-92c7-238aabf6a23e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126752025-04-30T06:38:45.301ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
126762025-04-30T06:38:45.303ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18a71d80-79bf-4c5b-9415-75d7f7a31f2a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126772025-04-30T06:38:45.304ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
126782025-04-30T06:38:45.309ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
126792025-04-30T06:38:45.309ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25ed5ca4-ab5e-45d5-962c-9f4dc1e8bc2d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126802025-04-30T06:38:45.310ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
126812025-04-30T06:38:45.319ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e93c5a17-8d0e-4ce4-a827-4f3109e4d9ce", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126822025-04-30T06:38:45.319ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
126832025-04-30T06:38:45.321ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4b92674-1ede-479b-8ece-649a1b45a442", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126842025-04-30T06:38:45.321ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
126852025-04-30T06:38:45.322ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9b38646-134a-44f0-b168-51850f6c5f6c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126862025-04-30T06:38:45.322ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
126872025-04-30T06:38:45.323ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a497e4f7-3b89-4838-9ec3-1942e6216f2c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126882025-04-30T06:38:45.323ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
126892025-04-30T06:38:45.324ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69ccbf82-9074-4820-89fb-e1fdd6210abb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126902025-04-30T06:38:45.324ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
126912025-04-30T06:38:45.325ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d58757aa-0d5a-43ae-9d57-b4ca1c55b287", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
126922025-04-30T06:38:45.325ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126932025-04-30T06:38:45.325ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126942025-04-30T06:38:45.325ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126952025-04-30T06:38:45.325ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126962025-04-30T06:38:45.326ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126972025-04-30T06:38:45.329ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126982025-04-30T06:38:45.333ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126992025-04-30T06:38:45.334ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
127002025-04-30T06:38:45.334ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127012025-04-30T06:38:45.334ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127022025-04-30T06:38:45.334ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127032025-04-30T06:38:45.343ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127042025-04-30T06:38:45.353ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127052025-04-30T06:38:45.359ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
127062025-04-30T06:38:45.359ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127072025-04-30T06:38:45.359ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127082025-04-30T06:38:45.360ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127092025-04-30T06:38:45.360ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127102025-04-30T06:38:45.360ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127112025-04-30T06:38:45.360ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127122025-04-30T06:38:45.360ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
127132025-04-30T06:38:45.361ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127142025-04-30T06:38:45.365ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127152025-04-30T06:38:45.371ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127162025-04-30T06:38:45.371ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127172025-04-30T06:38:45.371ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
127182025-04-30T06:38:45.372ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127192025-04-30T06:38:45.426ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
127202025-04-30T06:38:45.426ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
127212025-04-30T06:38:45.427ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:45535 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
127222025-04-30T06:38:45.428ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:45535 req_id = fadc71d3-0d9d-4593-8679-fdd18d6ff366 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
127232025-04-30T06:38:45.430ZINFOtest_quotas (dropshot): request completed latency_us = 2060 local_addr = [::1]:55655 method = GET remote_addr = [::1]:45535 req_id = fadc71d3-0d9d-4593-8679-fdd18d6ff366 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
127242025-04-30T06:38:45.430ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127252025-04-30T06:38:45.430ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
127262025-04-30T06:38:45.431ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127272025-04-30T06:38:45.432ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fadc71d3-0d9d-4593-8679-fdd18d6ff366", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
127282025-04-30T06:38:45.435ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
127292025-04-30T06:38:45.435ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
127302025-04-30T06:38:45.435ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
127312025-04-30T06:38:45.438ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
127322025-04-30T06:38:45.438ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127332025-04-30T06:38:45.438ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127342025-04-30T06:38:45.438ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127352025-04-30T06:38:45.438ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127362025-04-30T06:38:45.451ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127372025-04-30T06:38:45.457ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
127382025-04-30T06:38:45.459ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127392025-04-30T06:38:45.459ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127402025-04-30T06:38:45.463ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127412025-04-30T06:38:45.553ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
127422025-04-30T06:38:45.575ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
127432025-04-30T06:38:45.576ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
127442025-04-30T06:38:45.576ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 700.417176ms iteration = 45 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
127452025-04-30T06:38:45.576ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 46 reason = Signaled
127462025-04-30T06:38:45.576ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
127472025-04-30T06:38:45.576ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127482025-04-30T06:38:45.576ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127492025-04-30T06:38:45.576ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127502025-04-30T06:38:45.578ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127512025-04-30T06:38:45.582ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127522025-04-30T06:38:45.589ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
127532025-04-30T06:38:45.589ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127542025-04-30T06:38:45.589ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127552025-04-30T06:38:45.589ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127562025-04-30T06:38:45.596ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
127572025-04-30T06:38:45.596ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
127582025-04-30T06:38:45.596ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e3 21 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .!..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64933 req_id = 4bdba852-38f4-464e-a22e-7bd48a5c18ae
127592025-04-30T06:38:45.596ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58145,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64933 req_id = 4bdba852-38f4-464e-a22e-7bd48a5c18ae
127602025-04-30T06:38:45.597ZDEBGtest_quotas (store): query key key = _mgs._tcp
127612025-04-30T06:38:45.602ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127622025-04-30T06:38:45.602ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64933 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4bdba852-38f4-464e-a22e-7bd48a5c18ae
127632025-04-30T06:38:45.603ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3990, tv_nsec: 401808435 } })
127642025-04-30T06:38:45.603ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
127652025-04-30T06:38:45.603ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 70 f0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e p...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63298 req_id = dd862ff9-4959-443f-b427-0120c7ca3bb4
127662025-04-30T06:38:45.604ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28912,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63298 req_id = dd862ff9-4959-443f-b427-0120c7ca3bb4
127672025-04-30T06:38:45.604ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127682025-04-30T06:38:45.604ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63298 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dd862ff9-4959-443f-b427-0120c7ca3bb4
127692025-04-30T06:38:45.604ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127702025-04-30T06:38:45.605ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127712025-04-30T06:38:45.605ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127722025-04-30T06:38:45.605ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
127732025-04-30T06:38:45.608ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127742025-04-30T06:38:45.697ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
127752025-04-30T06:38:45.698ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c3 c3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61382 req_id = f390a311-f129-4452-8ed7-057cfb722669
127762025-04-30T06:38:45.698ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50115,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61382 req_id = f390a311-f129-4452-8ed7-057cfb722669
127772025-04-30T06:38:45.698ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
127782025-04-30T06:38:45.698ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50115, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61382 req_id = f390a311-f129-4452-8ed7-057cfb722669
127792025-04-30T06:38:45.699ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
127802025-04-30T06:38:45.699ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
127812025-04-30T06:38:45.699ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
127822025-04-30T06:38:45.701ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127832025-04-30T06:38:45.713ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127842025-04-30T06:38:45.713ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127852025-04-30T06:38:45.713ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
127862025-04-30T06:38:45.713ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127872025-04-30T06:38:45.715ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "723d4477-6be0-411a-9bad-c47780594cbf", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
127882025-04-30T06:38:45.716ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
127892025-04-30T06:38:45.721ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127902025-04-30T06:38:45.721ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
127912025-04-30T06:38:45.722ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127922025-04-30T06:38:45.723ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38daadea-9bb4-4044-9eba-5770388c74f1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
127932025-04-30T06:38:45.723ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
127942025-04-30T06:38:45.727ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "855a1cef-4a8f-49f2-9ad6-c7acf2435c0f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
127952025-04-30T06:38:45.727ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
127962025-04-30T06:38:45.730ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b4f6212-f820-4bb4-beb5-34abbb8c1aaf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
127972025-04-30T06:38:45.730ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
127982025-04-30T06:38:45.735ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f04de73-7f35-4900-bac9-82f792b12c0d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
127992025-04-30T06:38:45.735ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
128002025-04-30T06:38:45.737ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95bda3a4-1c92-45b5-82b6-4e1b53a82822", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128012025-04-30T06:38:45.737ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
128022025-04-30T06:38:45.739ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
128032025-04-30T06:38:45.739ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128042025-04-30T06:38:45.740ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 7eb76c82-fbf2-4244-9edc-c55cccf57c7d uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128052025-04-30T06:38:45.740ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 7eb76c82-fbf2-4244-9edc-c55cccf57c7d resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128062025-04-30T06:38:45.740ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 7eb76c82-fbf2-4244-9edc-c55cccf57c7d roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128072025-04-30T06:38:45.741ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 7eb76c82-fbf2-4244-9edc-c55cccf57c7d resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128082025-04-30T06:38:45.742ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe3b4797-45bf-4937-badb-9a886158c091", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128092025-04-30T06:38:45.742ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
128102025-04-30T06:38:45.744ZINFOtest_quotas (dropshot_internal): request completed latency_us = 4286 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 7eb76c82-fbf2-4244-9edc-c55cccf57c7d response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128112025-04-30T06:38:45.745ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eb76c82-fbf2-4244-9edc-c55cccf57c7d", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128122025-04-30T06:38:45.745ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128132025-04-30T06:38:45.745ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = cecd2f23-4cf8-4b31-bffb-327ba6f0e38a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128142025-04-30T06:38:45.746ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = cecd2f23-4cf8-4b31-bffb-327ba6f0e38a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128152025-04-30T06:38:45.746ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = cecd2f23-4cf8-4b31-bffb-327ba6f0e38a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128162025-04-30T06:38:45.746ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = cecd2f23-4cf8-4b31-bffb-327ba6f0e38a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128172025-04-30T06:38:45.750ZINFOtest_quotas (dropshot_internal): request completed latency_us = 4545 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = cecd2f23-4cf8-4b31-bffb-327ba6f0e38a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128182025-04-30T06:38:45.751ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "cecd2f23-4cf8-4b31-bffb-327ba6f0e38a", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128192025-04-30T06:38:45.751ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
128202025-04-30T06:38:45.751ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128212025-04-30T06:38:45.751ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
128222025-04-30T06:38:45.754ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bc2e918-ba5e-4d9f-aa80-5d581636a174", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128232025-04-30T06:38:45.755ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
128242025-04-30T06:38:45.756ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3696229-66cb-4461-b5a9-024a74064889", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128252025-04-30T06:38:45.756ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
128262025-04-30T06:38:45.758ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dcd4eed-9c20-48fb-91a8-4d13cb6c1bf4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128272025-04-30T06:38:45.758ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
128282025-04-30T06:38:45.762ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c468f969-a315-4b0a-a257-75706eedc3aa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128292025-04-30T06:38:45.762ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
128302025-04-30T06:38:45.762ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
128312025-04-30T06:38:45.764ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e160012f-ab3b-4842-a8b3-34504882818d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128322025-04-30T06:38:45.764ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
128332025-04-30T06:38:45.765ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fb7d8e1-28f2-4592-b92f-0469f763dc6b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128342025-04-30T06:38:45.766ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
128352025-04-30T06:38:45.768ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "497ebf21-ee35-4291-9190-e6b74a97a73f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128362025-04-30T06:38:45.768ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
128372025-04-30T06:38:45.770ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3ff157b-1c00-498f-ad34-13bbc9b5d027", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128382025-04-30T06:38:45.770ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
128392025-04-30T06:38:45.773ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "375bf35e-e8e0-4004-81e5-026e99a36645", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128402025-04-30T06:38:45.773ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
128412025-04-30T06:38:45.775ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56db8ac6-ff57-4c57-897d-c0aa072536a9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128422025-04-30T06:38:45.775ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
128432025-04-30T06:38:45.780ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d112a6aa-014b-443b-b204-b3089880d800", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128442025-04-30T06:38:45.780ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
128452025-04-30T06:38:45.782ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ec6dfd2-5cbb-4d38-91f2-9cbb90ea39af", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128462025-04-30T06:38:45.782ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
128472025-04-30T06:38:45.784ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92d6d006-95af-41dd-8283-4f8c0ee0bff5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128482025-04-30T06:38:45.784ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
128492025-04-30T06:38:45.785ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b294a19b-4a75-4b73-86a5-6eba61d689f1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128502025-04-30T06:38:45.785ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
128512025-04-30T06:38:45.786ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2de5ef4f-2a6a-4ab5-88ce-cd17d57ee32e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128522025-04-30T06:38:45.786ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
128532025-04-30T06:38:45.788ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "149b04d2-de7d-450d-adad-49bef2f3042e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128542025-04-30T06:38:45.788ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
128552025-04-30T06:38:45.789ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91b16b1b-cd9e-416d-98e8-a3f0fa203824", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128562025-04-30T06:38:45.789ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
128572025-04-30T06:38:45.792ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ebb5fe0-a205-4361-83e6-efc23022dcc1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128582025-04-30T06:38:45.792ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
128592025-04-30T06:38:45.794ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40d3ea4c-c3f9-42aa-b454-7a35844422ed", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128602025-04-30T06:38:45.795ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
128612025-04-30T06:38:45.796ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc04ad78-cd84-4750-a77b-0453ef7adbd3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128622025-04-30T06:38:45.797ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
128632025-04-30T06:38:45.799ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb1555e5-97e1-4700-8217-ebeb3f5b6a31", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128642025-04-30T06:38:45.800ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
128652025-04-30T06:38:45.802ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65a8750f-2c55-4671-b0a7-d3058a11f599", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128662025-04-30T06:38:45.802ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
128672025-04-30T06:38:45.805ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d67b4bde-6cd2-4585-940e-4e0e4c577577", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128682025-04-30T06:38:45.805ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
128692025-04-30T06:38:45.806ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e08a06a-dc8b-4bed-a967-9e44b365ffcf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128702025-04-30T06:38:45.807ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
128712025-04-30T06:38:45.809ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e294cab7-cf84-4ee6-ac48-79bc5abae350", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128722025-04-30T06:38:45.809ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
128732025-04-30T06:38:45.810ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3351918-15ca-4d83-8a90-716e76481aa0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128742025-04-30T06:38:45.810ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
128752025-04-30T06:38:45.814ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128762025-04-30T06:38:45.814ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128772025-04-30T06:38:45.814ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128782025-04-30T06:38:45.814ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
128792025-04-30T06:38:45.815ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128802025-04-30T06:38:45.819ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1412d9c2-432a-49ea-a086-fdbb6390f96b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128812025-04-30T06:38:45.820ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
128822025-04-30T06:38:45.822ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d510315f-444d-4dd8-9d4f-9bb335280aec", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128832025-04-30T06:38:45.823ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
128842025-04-30T06:38:45.823ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128852025-04-30T06:38:45.832ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128862025-04-30T06:38:45.832ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128872025-04-30T06:38:45.832ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
128882025-04-30T06:38:45.833ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128892025-04-30T06:38:45.844ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128902025-04-30T06:38:45.844ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
128912025-04-30T06:38:45.845ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128922025-04-30T06:38:45.859ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128932025-04-30T06:38:45.859ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eae63c8-1b60-4db0-b6e9-4961fa282e7e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
128942025-04-30T06:38:45.860ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
128952025-04-30T06:38:45.860ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128962025-04-30T06:38:45.864ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128972025-04-30T06:38:45.866ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128982025-04-30T06:38:45.867ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128992025-04-30T06:38:45.867ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129002025-04-30T06:38:45.871ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df32708e-6af6-4dc0-8bd6-9926e9765837", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
129012025-04-30T06:38:45.871ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
129022025-04-30T06:38:45.877ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef4467a8-4233-48a7-a2cc-5d3c3016ef84", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
129032025-04-30T06:38:45.878ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
129042025-04-30T06:38:45.883ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa487914-a9e9-4949-b650-1a4faf9869bf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
129052025-04-30T06:38:45.884ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
129062025-04-30T06:38:45.895ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96d24125-fadd-4e7d-9047-7d8f51770787", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
129072025-04-30T06:38:45.896ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
129082025-04-30T06:38:45.903ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e603e31b-5d7b-4c2a-99dd-4f420b37ccc2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
129092025-04-30T06:38:45.903ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
129102025-04-30T06:38:45.904ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
129112025-04-30T06:38:45.909ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59bfdbc2-461e-445f-8687-dc21d8872816", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
129122025-04-30T06:38:45.909ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
129132025-04-30T06:38:45.916ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bd8d93e-2f8f-45d4-a549-164d4ad85bfe", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
129142025-04-30T06:38:45.917ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
129152025-04-30T06:38:45.918ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56babc20-4dfe-4f90-a9c8-91f72057cf71", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
129162025-04-30T06:38:45.928ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
129172025-04-30T06:38:45.930ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:47306 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
129182025-04-30T06:38:45.931ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:47306 req_id = aec72329-7b94-4077-bd11-ba0f4a284be3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
129192025-04-30T06:38:45.932ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 590 local_addr = [::1]:40067 method = GET remote_addr = [::1]:47306 req_id = aec72329-7b94-4077-bd11-ba0f4a284be3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
129202025-04-30T06:38:45.935ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32aecc7e-36de-4f81-9689-42451393691d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
129212025-04-30T06:38:45.935ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
129222025-04-30T06:38:45.935ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129232025-04-30T06:38:45.935ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129242025-04-30T06:38:45.935ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129252025-04-30T06:38:45.936ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129262025-04-30T06:38:45.941ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129272025-04-30T06:38:45.947ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
129282025-04-30T06:38:45.947ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
129292025-04-30T06:38:45.947ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129302025-04-30T06:38:45.947ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129312025-04-30T06:38:45.947ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129322025-04-30T06:38:45.948ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129332025-04-30T06:38:45.957ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129342025-04-30T06:38:45.963ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
129352025-04-30T06:38:45.963ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129362025-04-30T06:38:45.963ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129372025-04-30T06:38:45.965ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129382025-04-30T06:38:45.967ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129392025-04-30T06:38:45.967ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129402025-04-30T06:38:45.967ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129412025-04-30T06:38:45.967ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
129422025-04-30T06:38:45.968ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129432025-04-30T06:38:46.042ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
129442025-04-30T06:38:46.042ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
129452025-04-30T06:38:46.044ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:38946 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
129462025-04-30T06:38:46.046ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129472025-04-30T06:38:46.058ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129482025-04-30T06:38:46.058ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129492025-04-30T06:38:46.058ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
129502025-04-30T06:38:46.059ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129512025-04-30T06:38:46.060ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:38946 req_id = 5a75729f-c704-46ad-a590-a50a957401d5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
129522025-04-30T06:38:46.065ZINFOtest_quotas (dropshot): request completed latency_us = 5389 local_addr = [::1]:55655 method = GET remote_addr = [::1]:38946 req_id = 5a75729f-c704-46ad-a590-a50a957401d5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
129532025-04-30T06:38:46.066ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a75729f-c704-46ad-a590-a50a957401d5", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:45 GMT"} })
129542025-04-30T06:38:46.072ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
129552025-04-30T06:38:46.073ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
129562025-04-30T06:38:46.073ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
129572025-04-30T06:38:46.078ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
129582025-04-30T06:38:46.078ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129592025-04-30T06:38:46.078ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129602025-04-30T06:38:46.078ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129612025-04-30T06:38:46.080ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129622025-04-30T06:38:46.082ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129632025-04-30T06:38:46.082ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
129642025-04-30T06:38:46.082ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129652025-04-30T06:38:46.107ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
129662025-04-30T06:38:46.107ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
129672025-04-30T06:38:46.107ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
129682025-04-30T06:38:46.110ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
129692025-04-30T06:38:46.111ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129702025-04-30T06:38:46.117ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
129712025-04-30T06:38:46.117ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129722025-04-30T06:38:46.117ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129732025-04-30T06:38:46.118ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129742025-04-30T06:38:46.122ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
129752025-04-30T06:38:46.123ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
129762025-04-30T06:38:46.131ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
129772025-04-30T06:38:46.131ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
129782025-04-30T06:38:46.182ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
129792025-04-30T06:38:46.192ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
129802025-04-30T06:38:46.192ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
129812025-04-30T06:38:46.192ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 616.019368ms iteration = 46 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
129822025-04-30T06:38:46.192ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 47 reason = Signaled
129832025-04-30T06:38:46.193ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
129842025-04-30T06:38:46.193ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129852025-04-30T06:38:46.193ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129862025-04-30T06:38:46.193ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129872025-04-30T06:38:46.193ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129882025-04-30T06:38:46.197ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129892025-04-30T06:38:46.203ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
129902025-04-30T06:38:46.203ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129912025-04-30T06:38:46.203ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129922025-04-30T06:38:46.204ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129932025-04-30T06:38:46.209ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
129942025-04-30T06:38:46.217ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
129952025-04-30T06:38:46.217ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
129962025-04-30T06:38:46.218ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 24 6d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 $m..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56704 req_id = cea6501f-7699-493d-94d9-6d7f29d73cbf
129972025-04-30T06:38:46.218ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9325,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56704 req_id = cea6501f-7699-493d-94d9-6d7f29d73cbf
129982025-04-30T06:38:46.218ZDEBGtest_quotas (store): query key key = _mgs._tcp
129992025-04-30T06:38:46.218ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130002025-04-30T06:38:46.218ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56704 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cea6501f-7699-493d-94d9-6d7f29d73cbf
130012025-04-30T06:38:46.219ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3991, tv_nsec: 17978013 } })
130022025-04-30T06:38:46.219ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
130032025-04-30T06:38:46.219ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 93 2a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .*..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64990 req_id = 60e3ccc5-de4e-433c-8c6c-71d9da536e85
130042025-04-30T06:38:46.219ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37674,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64990 req_id = 60e3ccc5-de4e-433c-8c6c-71d9da536e85
130052025-04-30T06:38:46.219ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130062025-04-30T06:38:46.219ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64990 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 60e3ccc5-de4e-433c-8c6c-71d9da536e85
130072025-04-30T06:38:46.281ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
130082025-04-30T06:38:46.282ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130092025-04-30T06:38:46.282ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130102025-04-30T06:38:46.282ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130112025-04-30T06:38:46.282ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
130122025-04-30T06:38:46.283ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130132025-04-30T06:38:46.284ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 00 08 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56779 req_id = b96f2020-368a-4599-8539-c711c55c0d29
130142025-04-30T06:38:46.284ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56779 req_id = b96f2020-368a-4599-8539-c711c55c0d29
130152025-04-30T06:38:46.285ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
130162025-04-30T06:38:46.285ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56779 req_id = b96f2020-368a-4599-8539-c711c55c0d29
130172025-04-30T06:38:46.285ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
130182025-04-30T06:38:46.285ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
130192025-04-30T06:38:46.285ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
130202025-04-30T06:38:46.289ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130212025-04-30T06:38:46.299ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130222025-04-30T06:38:46.299ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130232025-04-30T06:38:46.299ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
130242025-04-30T06:38:46.299ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130252025-04-30T06:38:46.302ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6bac4e4-ebd8-4ec9-a774-4f4a39bb9b81", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130262025-04-30T06:38:46.302ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
130272025-04-30T06:38:46.303ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130282025-04-30T06:38:46.303ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
130292025-04-30T06:38:46.303ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130302025-04-30T06:38:46.305ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e917afb6-b82c-4734-9bf5-810fcbf86ba6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130312025-04-30T06:38:46.305ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
130322025-04-30T06:38:46.320ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "619ca61b-99b6-4fb9-bbf7-0030bf05ecdd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130332025-04-30T06:38:46.320ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
130342025-04-30T06:38:46.327ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90a2c40f-b751-432e-a226-6152abcedac7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130352025-04-30T06:38:46.327ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
130362025-04-30T06:38:46.332ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9172ab22-b4dd-402b-9d73-f7afbdbf8423", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130372025-04-30T06:38:46.332ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
130382025-04-30T06:38:46.335ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:44732 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130392025-04-30T06:38:46.337ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:44732 req_id = 4a95008d-9246-42ee-8aae-3d677bbfcb66 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130402025-04-30T06:38:46.338ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 837 local_addr = [::1]:40067 method = GET remote_addr = [::1]:44732 req_id = 4a95008d-9246-42ee-8aae-3d677bbfcb66 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130412025-04-30T06:38:46.351ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:54103 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130422025-04-30T06:38:46.353ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:50453 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130432025-04-30T06:38:46.354ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
130442025-04-30T06:38:46.354ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:56335 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130452025-04-30T06:38:46.355ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:40814 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130462025-04-30T06:38:46.356ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:54103 req_id = 0f133a03-754d-4f21-abcc-34d7d9475347 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130472025-04-30T06:38:46.356ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:50453 req_id = 578dc67d-de94-44eb-b182-6293fa01fe9b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130482025-04-30T06:38:46.360ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 4706 local_addr = [::1]:40067 method = GET remote_addr = [::1]:54103 req_id = 0f133a03-754d-4f21-abcc-34d7d9475347 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130492025-04-30T06:38:46.360ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 4756 local_addr = [::1]:40067 method = GET remote_addr = [::1]:50453 req_id = 578dc67d-de94-44eb-b182-6293fa01fe9b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130502025-04-30T06:38:46.363ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:59923 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130512025-04-30T06:38:46.363ZINFOtest_quotas (dropshot): accepted connection kind = pantry local_addr = [::1]:40067 remote_addr = [::1]:37120 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130522025-04-30T06:38:46.364ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:56335 req_id = 952efd51-c5db-40dc-8acb-4d803cf9e94f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130532025-04-30T06:38:46.364ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:40814 req_id = dec0e4fb-afaa-4cae-bc86-487cf1da541c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130542025-04-30T06:38:46.365ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1169 local_addr = [::1]:40067 method = GET remote_addr = [::1]:56335 req_id = 952efd51-c5db-40dc-8acb-4d803cf9e94f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130552025-04-30T06:38:46.366ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1178 local_addr = [::1]:40067 method = GET remote_addr = [::1]:40814 req_id = dec0e4fb-afaa-4cae-bc86-487cf1da541c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130562025-04-30T06:38:46.376ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:59923 req_id = a3fd6494-b157-4f58-b8e8-fb003d99cef6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130572025-04-30T06:38:46.376ZTRACtest_quotas (dropshot): incoming request kind = pantry local_addr = [::1]:40067 method = GET remote_addr = [::1]:37120 req_id = e6baae7f-d496-475e-9a70-a026fc8f1023 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130582025-04-30T06:38:46.377ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1171 local_addr = [::1]:40067 method = GET remote_addr = [::1]:59923 req_id = a3fd6494-b157-4f58-b8e8-fb003d99cef6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130592025-04-30T06:38:46.377ZINFOtest_quotas (dropshot): request completed kind = pantry latency_us = 1078 local_addr = [::1]:40067 method = GET remote_addr = [::1]:37120 req_id = e6baae7f-d496-475e-9a70-a026fc8f1023 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
130602025-04-30T06:38:46.378ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b9f1703-2823-48a6-a8c0-d1e30df8c0fb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130612025-04-30T06:38:46.379ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
130622025-04-30T06:38:46.384ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe9eb86b-4cc2-4e39-a8ba-41f222827b17", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130632025-04-30T06:38:46.385ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
130642025-04-30T06:38:46.391ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f01c1cd-8eb3-4f47-bcb3-9ad46c4e76cf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130652025-04-30T06:38:46.391ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
130662025-04-30T06:38:46.392ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8937223-dd38-48e2-a62a-05534624e865", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130672025-04-30T06:38:46.393ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
130682025-04-30T06:38:46.394ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a62944d-c38e-4587-8dbf-460439063dff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130692025-04-30T06:38:46.394ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
130702025-04-30T06:38:46.396ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca7efa5e-c57e-4f8c-ad1e-2e7a72d36a27", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130712025-04-30T06:38:46.397ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
130722025-04-30T06:38:46.399ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd61d71-b9aa-45cf-b617-dc098e895f8a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130732025-04-30T06:38:46.399ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
130742025-04-30T06:38:46.401ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7151e0c-a43a-45f0-b39d-0caa880ea517", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130752025-04-30T06:38:46.401ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
130762025-04-30T06:38:46.410ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130772025-04-30T06:38:46.411ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130782025-04-30T06:38:46.411ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130792025-04-30T06:38:46.411ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
130802025-04-30T06:38:46.411ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130812025-04-30T06:38:46.412ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f29d8f54-505c-4c40-8294-df46bb4ca786", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130822025-04-30T06:38:46.412ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
130832025-04-30T06:38:46.415ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130842025-04-30T06:38:46.422ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130852025-04-30T06:38:46.423ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130862025-04-30T06:38:46.423ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
130872025-04-30T06:38:46.424ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130882025-04-30T06:38:46.431ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130892025-04-30T06:38:46.431ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
130902025-04-30T06:38:46.431ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130912025-04-30T06:38:46.432ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6b58c7f-186e-4ef9-ba03-0653b011b1cd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130922025-04-30T06:38:46.432ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
130932025-04-30T06:38:46.434ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89fac640-0daf-4e08-97f5-cb7298785527", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130942025-04-30T06:38:46.434ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
130952025-04-30T06:38:46.448ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "100067b0-1239-4d53-839b-0e1e68ad54af", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130962025-04-30T06:38:46.449ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
130972025-04-30T06:38:46.453ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06ea65dc-8cf5-4cea-939c-b1332512c139", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
130982025-04-30T06:38:46.453ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
130992025-04-30T06:38:46.457ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b842096-55da-4881-bdda-07c22606ea16", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131002025-04-30T06:38:46.457ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
131012025-04-30T06:38:46.459ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef37226f-6924-42ae-a9af-4cfb1ee20683", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131022025-04-30T06:38:46.459ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
131032025-04-30T06:38:46.460ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e7273ad-5e58-4173-8b46-755c0ba41c64", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131042025-04-30T06:38:46.461ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
131052025-04-30T06:38:46.463ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d3789b3-bb65-4039-9f5a-3102ffe8998a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131062025-04-30T06:38:46.463ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
131072025-04-30T06:38:46.471ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23118f13-b0b9-46f7-9848-648fee431598", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131082025-04-30T06:38:46.472ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
131092025-04-30T06:38:46.472ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
131102025-04-30T06:38:46.476ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "179de735-9029-4c54-8d5e-accba634fbea", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131112025-04-30T06:38:46.476ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
131122025-04-30T06:38:46.479ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2b470c0-09f7-46b4-bec9-8688852554a5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131132025-04-30T06:38:46.480ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
131142025-04-30T06:38:46.482ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "912249da-aa61-460f-b96c-ca920d5ae223", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131152025-04-30T06:38:46.482ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
131162025-04-30T06:38:46.484ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63e38db7-d547-41e8-b86d-7c291dc98c49", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131172025-04-30T06:38:46.484ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
131182025-04-30T06:38:46.486ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39a6228d-745a-448f-96dc-7b5f419672d0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131192025-04-30T06:38:46.486ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
131202025-04-30T06:38:46.490ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac15f360-ccfb-4a8d-9229-88b40d5342cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131212025-04-30T06:38:46.490ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
131222025-04-30T06:38:46.494ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9b904fa-a1f2-43b9-a1d9-3e7ea05358bc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131232025-04-30T06:38:46.494ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
131242025-04-30T06:38:46.496ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7fd0997-feb4-4162-aed8-5a8320c151dd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131252025-04-30T06:38:46.499ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
131262025-04-30T06:38:46.501ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "528ac14e-5ad9-424e-a257-559eca35df82", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131272025-04-30T06:38:46.501ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
131282025-04-30T06:38:46.503ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbd95c3a-163a-4e9e-a187-2fabac940969", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131292025-04-30T06:38:46.503ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
131302025-04-30T06:38:46.505ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0cb0c7c-bc38-4264-8034-8ca537b2331a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131312025-04-30T06:38:46.506ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
131322025-04-30T06:38:46.508ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a84509fe-4661-4353-b277-d3703fec9100", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131332025-04-30T06:38:46.508ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
131342025-04-30T06:38:46.511ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c0c8b20-30d3-40cf-b34e-0c108dfb895a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131352025-04-30T06:38:46.511ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
131362025-04-30T06:38:46.514ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "804fc444-ccbb-4490-9439-e4bc2ba9c141", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131372025-04-30T06:38:46.514ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
131382025-04-30T06:38:46.520ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7722cc8d-4be8-421f-92d3-1ae561baaffa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131392025-04-30T06:38:46.520ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
131402025-04-30T06:38:46.523ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ffc2c01-a197-4c8c-8408-b1f2060e7511", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131412025-04-30T06:38:46.523ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
131422025-04-30T06:38:46.524ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131432025-04-30T06:38:46.524ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131442025-04-30T06:38:46.524ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131452025-04-30T06:38:46.524ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
131462025-04-30T06:38:46.525ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131472025-04-30T06:38:46.527ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3123d32a-6223-424b-b5a8-ce53fd9a62ba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131482025-04-30T06:38:46.527ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
131492025-04-30T06:38:46.529ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131502025-04-30T06:38:46.539ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131512025-04-30T06:38:46.540ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131522025-04-30T06:38:46.540ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
131532025-04-30T06:38:46.540ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131542025-04-30T06:38:46.542ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29c3196d-2737-49cb-b78c-e77435698050", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131552025-04-30T06:38:46.543ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
131562025-04-30T06:38:46.546ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131572025-04-30T06:38:46.546ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
131582025-04-30T06:38:46.547ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131592025-04-30T06:38:46.547ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26ced15e-7039-4e7b-8496-964ed406eed6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131602025-04-30T06:38:46.548ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
131612025-04-30T06:38:46.551ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75a1aa50-d4ed-473a-aa73-5cd4a3c144aa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131622025-04-30T06:38:46.551ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
131632025-04-30T06:38:46.552ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e712cdf-01fa-4ce4-a814-3b8cdd96ee63", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131642025-04-30T06:38:46.553ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
131652025-04-30T06:38:46.554ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e659de9a-8082-4743-96e2-e8dd51a95ac6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131662025-04-30T06:38:46.554ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
131672025-04-30T06:38:46.554ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131682025-04-30T06:38:46.554ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131692025-04-30T06:38:46.554ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131702025-04-30T06:38:46.555ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131712025-04-30T06:38:46.558ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131722025-04-30T06:38:46.564ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
131732025-04-30T06:38:46.564ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
131742025-04-30T06:38:46.564ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131752025-04-30T06:38:46.565ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131762025-04-30T06:38:46.565ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131772025-04-30T06:38:46.565ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131782025-04-30T06:38:46.575ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131792025-04-30T06:38:46.583ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
131802025-04-30T06:38:46.583ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131812025-04-30T06:38:46.583ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131822025-04-30T06:38:46.584ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131832025-04-30T06:38:46.688ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
131842025-04-30T06:38:46.688ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
131852025-04-30T06:38:46.689ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:46340 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
131862025-04-30T06:38:46.723ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:46340 req_id = 91c021dc-fe95-4185-b020-10e7e0564f75 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
131872025-04-30T06:38:46.724ZINFOtest_quotas (dropshot): request completed latency_us = 1146 local_addr = [::1]:55655 method = GET remote_addr = [::1]:46340 req_id = 91c021dc-fe95-4185-b020-10e7e0564f75 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
131882025-04-30T06:38:46.726ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "91c021dc-fe95-4185-b020-10e7e0564f75", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
131892025-04-30T06:38:46.729ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131902025-04-30T06:38:46.730ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131912025-04-30T06:38:46.730ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
131922025-04-30T06:38:46.736ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131932025-04-30T06:38:46.736ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131942025-04-30T06:38:46.736ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131952025-04-30T06:38:46.736ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131962025-04-30T06:38:46.737ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131972025-04-30T06:38:46.742ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131982025-04-30T06:38:46.747ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131992025-04-30T06:38:46.747ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132002025-04-30T06:38:46.748ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132012025-04-30T06:38:46.748ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132022025-04-30T06:38:46.757ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
132032025-04-30T06:38:46.777ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
132042025-04-30T06:38:46.778ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 3a 63 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 :c..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63583 req_id = 297599d2-a16f-4da0-a6e3-cd83240da93d
132052025-04-30T06:38:46.778ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14947,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63583 req_id = 297599d2-a16f-4da0-a6e3-cd83240da93d
132062025-04-30T06:38:46.778ZDEBGtest_quotas (store): query key key = _dendrite._tcp
132072025-04-30T06:38:46.778ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132082025-04-30T06:38:46.778ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63583 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 297599d2-a16f-4da0-a6e3-cd83240da93d
132092025-04-30T06:38:46.779ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3991, tv_nsec: 578254136 } })
132102025-04-30T06:38:46.779ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
132112025-04-30T06:38:46.780ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 72 d3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e r...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57325 req_id = eff7f504-0e2b-4404-9bd2-18e95c09b949
132122025-04-30T06:38:46.780ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29395,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57325 req_id = eff7f504-0e2b-4404-9bd2-18e95c09b949
132132025-04-30T06:38:46.780ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132142025-04-30T06:38:46.780ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57325 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = eff7f504-0e2b-4404-9bd2-18e95c09b949
132152025-04-30T06:38:46.780ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
132162025-04-30T06:38:46.831ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
132172025-04-30T06:38:46.831ZDEBGtest_quotas (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
132182025-04-30T06:38:46.831ZDEBGtest_quotas (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
132192025-04-30T06:38:46.831ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
132202025-04-30T06:38:46.835ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
132212025-04-30T06:38:46.835ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
132222025-04-30T06:38:46.835ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132232025-04-30T06:38:46.837ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132242025-04-30T06:38:46.838ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
132252025-04-30T06:38:46.838ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132262025-04-30T06:38:46.838ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
132272025-04-30T06:38:46.838ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 73 8c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 s............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49626 req_id = 400a3463-473a-4311-953c-8e523757fa90
132282025-04-30T06:38:46.838ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29580,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49626 req_id = 400a3463-473a-4311-953c-8e523757fa90
132292025-04-30T06:38:46.838ZDEBGtest_quotas (store): query key key = _dendrite._tcp
132302025-04-30T06:38:46.838ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132312025-04-30T06:38:46.838ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49626 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 400a3463-473a-4311-953c-8e523757fa90
132322025-04-30T06:38:46.840ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3991, tv_nsec: 638955159 } })
132332025-04-30T06:38:46.840ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
132342025-04-30T06:38:46.840ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132352025-04-30T06:38:46.847ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132362025-04-30T06:38:46.847ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132372025-04-30T06:38:46.847ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
132382025-04-30T06:38:46.847ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132392025-04-30T06:38:46.851ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132402025-04-30T06:38:46.851ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 97 ee 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49647 req_id = dba6a5ca-ec02-4f9f-b3a1-9cc3e3a845f1
132412025-04-30T06:38:46.851ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38894,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49647 req_id = dba6a5ca-ec02-4f9f-b3a1-9cc3e3a845f1
132422025-04-30T06:38:46.853ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132432025-04-30T06:38:46.853ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49647 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dba6a5ca-ec02-4f9f-b3a1-9cc3e3a845f1
132442025-04-30T06:38:46.853ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132452025-04-30T06:38:46.854ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
132462025-04-30T06:38:46.917ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
132472025-04-30T06:38:46.917ZDEBGtest_quotas (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
132482025-04-30T06:38:46.917ZDEBGtest_quotas (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
132492025-04-30T06:38:46.918ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
132502025-04-30T06:38:46.919ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
132512025-04-30T06:38:46.942ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132522025-04-30T06:38:46.943ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132532025-04-30T06:38:46.944ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132542025-04-30T06:38:46.944ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132552025-04-30T06:38:46.944ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
132562025-04-30T06:38:46.945ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
132572025-04-30T06:38:46.945ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 752.628042ms iteration = 47 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
132582025-04-30T06:38:46.945ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 48 reason = Signaled
132592025-04-30T06:38:46.945ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
132602025-04-30T06:38:46.945ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132612025-04-30T06:38:46.947ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132622025-04-30T06:38:46.947ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132632025-04-30T06:38:46.948ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132642025-04-30T06:38:46.952ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132652025-04-30T06:38:46.952ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
132662025-04-30T06:38:46.952ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132672025-04-30T06:38:46.953ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132682025-04-30T06:38:46.958ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
132692025-04-30T06:38:46.958ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
132702025-04-30T06:38:46.958ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
132712025-04-30T06:38:46.959ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
132722025-04-30T06:38:46.967ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
132732025-04-30T06:38:46.967ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
132742025-04-30T06:38:46.973ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f3 57 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .W..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49766 req_id = 848e6c21-d3be-4582-aa8b-0dffd786cbf0
132752025-04-30T06:38:46.974ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62295,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49766 req_id = 848e6c21-d3be-4582-aa8b-0dffd786cbf0
132762025-04-30T06:38:46.974ZDEBGtest_quotas (store): query key key = _mgs._tcp
132772025-04-30T06:38:46.974ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132782025-04-30T06:38:46.974ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49766 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 848e6c21-d3be-4582-aa8b-0dffd786cbf0
132792025-04-30T06:38:46.974ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3991, tv_nsec: 773650711 } })
132802025-04-30T06:38:46.974ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
132812025-04-30T06:38:46.980ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d0 ea 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60603 req_id = 1d46ef46-b56e-4636-9cb7-7d94eb5c4f37
132822025-04-30T06:38:46.980ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53482,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60603 req_id = 1d46ef46-b56e-4636-9cb7-7d94eb5c4f37
132832025-04-30T06:38:46.980ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132842025-04-30T06:38:46.980ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60603 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1d46ef46-b56e-4636-9cb7-7d94eb5c4f37
132852025-04-30T06:38:47.045ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
132862025-04-30T06:38:47.049ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ba e6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52662 req_id = f12287f5-a48a-4e6e-8ac4-a671363a6ed7
132872025-04-30T06:38:47.049ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47846,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52662 req_id = f12287f5-a48a-4e6e-8ac4-a671363a6ed7
132882025-04-30T06:38:47.049ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
132892025-04-30T06:38:47.049ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47846, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52662 req_id = f12287f5-a48a-4e6e-8ac4-a671363a6ed7
132902025-04-30T06:38:47.050ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
132912025-04-30T06:38:47.059ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
132922025-04-30T06:38:47.060ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
132932025-04-30T06:38:47.092ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c58e12b-1902-4647-90f4-74679b8e08b6", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
132942025-04-30T06:38:47.093ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
132952025-04-30T06:38:47.098ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f251349-6ab4-41cb-aba1-0dbf02ba8ddd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
132962025-04-30T06:38:47.098ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
132972025-04-30T06:38:47.101ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "baea0bf1-f590-4765-ad64-93263f57ec51", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
132982025-04-30T06:38:47.103ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
132992025-04-30T06:38:47.109ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
133002025-04-30T06:38:47.109ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
133012025-04-30T06:38:47.110ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
133022025-04-30T06:38:47.112ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
133032025-04-30T06:38:47.114ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
133042025-04-30T06:38:47.118ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
133052025-04-30T06:38:47.118ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
133062025-04-30T06:38:47.120ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a64d1be7-6828-4c63-ba03-f4d46635938a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133072025-04-30T06:38:47.121ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
133082025-04-30T06:38:47.122ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
133092025-04-30T06:38:47.122ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
133102025-04-30T06:38:47.123ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82a0c7ad-cc81-41b8-ac90-72a484327187", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133112025-04-30T06:38:47.123ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
133122025-04-30T06:38:47.126ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37a50f60-75ae-41ef-bb15-d0e80a4854c0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133132025-04-30T06:38:47.126ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
133142025-04-30T06:38:47.132ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dd572df-831c-489e-a6c2-92437a56d03b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133152025-04-30T06:38:47.135ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
133162025-04-30T06:38:47.139ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30860a5f-df83-4bc4-b995-ab8024fca6c1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133172025-04-30T06:38:47.140ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
133182025-04-30T06:38:47.145ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0058a50-9239-420c-95c9-8336df0fef1a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133192025-04-30T06:38:47.145ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
133202025-04-30T06:38:47.147ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "afb79145-0190-4a82-a8f4-c32281a5cf66", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133212025-04-30T06:38:47.148ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
133222025-04-30T06:38:47.149ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc0ed39b-454e-487a-b468-f1515ded0d56", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133232025-04-30T06:38:47.149ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
133242025-04-30T06:38:47.150ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6a9a6e6-d708-407f-b257-76a2237b32d8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133252025-04-30T06:38:47.150ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
133262025-04-30T06:38:47.152ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a8e315c-af28-4a22-bad7-3ff9b0efa6ab", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133272025-04-30T06:38:47.153ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
133282025-04-30T06:38:47.155ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "512b9e73-d8cb-40e4-8557-738faa3bdc1d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133292025-04-30T06:38:47.155ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
133302025-04-30T06:38:47.157ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f3fb9ba-9573-4554-b705-ac88b1acc6ba", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133312025-04-30T06:38:47.158ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
133322025-04-30T06:38:47.159ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b9c1962-eb47-4f41-9345-9f9f579b6f04", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133332025-04-30T06:38:47.159ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
133342025-04-30T06:38:47.161ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "acf975df-8e1e-4a69-85dd-59800a67a011", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133352025-04-30T06:38:47.161ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
133362025-04-30T06:38:47.164ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "567208b8-3b22-498f-a1bd-16a15b298949", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133372025-04-30T06:38:47.164ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
133382025-04-30T06:38:47.165ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133392025-04-30T06:38:47.165ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133402025-04-30T06:38:47.165ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133412025-04-30T06:38:47.165ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
133422025-04-30T06:38:47.166ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133432025-04-30T06:38:47.169ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133442025-04-30T06:38:47.177ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133452025-04-30T06:38:47.177ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133462025-04-30T06:38:47.177ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
133472025-04-30T06:38:47.178ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133482025-04-30T06:38:47.179ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0cec4b6-ea17-40e2-83e6-c0a68b2aed2c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133492025-04-30T06:38:47.179ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
133502025-04-30T06:38:47.183ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133512025-04-30T06:38:47.183ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
133522025-04-30T06:38:47.185ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133532025-04-30T06:38:47.187ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61f0bc65-6cdc-45cf-b340-98d85c87fab5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133542025-04-30T06:38:47.187ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
133552025-04-30T06:38:47.188ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5cd9116-86a3-4b05-a489-256ac6ef686a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133562025-04-30T06:38:47.189ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
133572025-04-30T06:38:47.191ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cc657cb-fdb4-4005-a1be-19d8e9389398", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133582025-04-30T06:38:47.191ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
133592025-04-30T06:38:47.193ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "307afbc0-5495-4e12-848a-5ba462b47b46", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133602025-04-30T06:38:47.194ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
133612025-04-30T06:38:47.194ZTRACtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
133622025-04-30T06:38:47.196ZTRACtest_quotas (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 1f 07 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61587 req_id = 0c95d560-0e75-4129-8ea3-03ce275bba83
133632025-04-30T06:38:47.196ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7943,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61587 req_id = 0c95d560-0e75-4129-8ea3-03ce275bba83
133642025-04-30T06:38:47.196ZDEBGtest_quotas (store): query key key = _dendrite._tcp
133652025-04-30T06:38:47.196ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133662025-04-30T06:38:47.196ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61587 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0c95d560-0e75-4129-8ea3-03ce275bba83
133672025-04-30T06:38:47.200ZDEBGtest_quotas (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55999, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3991, tv_nsec: 999871591 } })
133682025-04-30T06:38:47.201ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
133692025-04-30T06:38:47.210ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 40 eb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e @...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61673 req_id = a3320a3e-686d-4217-93a6-3acff5f9fabd
133702025-04-30T06:38:47.210ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16619,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61673 req_id = a3320a3e-686d-4217-93a6-3acff5f9fabd
133712025-04-30T06:38:47.210ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133722025-04-30T06:38:47.210ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61673 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a3320a3e-686d-4217-93a6-3acff5f9fabd
133732025-04-30T06:38:47.211ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e381b4b2-780a-48c4-876f-c47e189b3d1c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:46 GMT"} })
133742025-04-30T06:38:47.213ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
133752025-04-30T06:38:47.214ZINFOtest_quotas (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
133762025-04-30T06:38:47.285ZINFOtest_quotas (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
133772025-04-30T06:38:47.285ZDEBGtest_quotas (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
133782025-04-30T06:38:47.286ZDEBGtest_quotas (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
133792025-04-30T06:38:47.286ZWARNtest_quotas (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
133802025-04-30T06:38:47.287ZWARNtest_quotas (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
133812025-04-30T06:38:47.287ZWARNtest_quotas (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
133822025-04-30T06:38:47.293ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d1fcb31-05a5-45d1-8d76-5445ff711075", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
133832025-04-30T06:38:47.294ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
133842025-04-30T06:38:47.300ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30683087-5b5b-4463-a04c-63935ec750d9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
133852025-04-30T06:38:47.301ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
133862025-04-30T06:38:47.309ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
133872025-04-30T06:38:47.310ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce5ffed6-4101-49dd-af04-e5b153e7a90e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
133882025-04-30T06:38:47.310ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
133892025-04-30T06:38:47.315ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b75c0f3-46b8-4d6b-9d8d-fe5ab12d7e4a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
133902025-04-30T06:38:47.315ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
133912025-04-30T06:38:47.320ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6c90534-155d-43a5-84a6-7c00da3d2159", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
133922025-04-30T06:38:47.320ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
133932025-04-30T06:38:47.323ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2edfcfe4-7662-4551-9d8e-340990ebb8e8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
133942025-04-30T06:38:47.323ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
133952025-04-30T06:38:47.324ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2eb160e-7b79-4d6b-80dc-0a2a939f4b3f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
133962025-04-30T06:38:47.324ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
133972025-04-30T06:38:47.325ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2845dd5-ebe7-450d-8289-badcce32e3c5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
133982025-04-30T06:38:47.332ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
133992025-04-30T06:38:47.333ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d439d24-8da6-443a-b935-7f8bbf296cbe", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134002025-04-30T06:38:47.333ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
134012025-04-30T06:38:47.335ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4bc14e3-9447-405b-aa97-86449a8d288d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134022025-04-30T06:38:47.335ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
134032025-04-30T06:38:47.337ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7932bf0a-9f05-4ab9-837c-39808b2f445a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134042025-04-30T06:38:47.338ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
134052025-04-30T06:38:47.340ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0a8abcc-b347-43b4-bdb5-78b2e06d22c6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134062025-04-30T06:38:47.341ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
134072025-04-30T06:38:47.342ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76b8d5f9-5fbb-4999-81ad-0ff73968ff74", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134082025-04-30T06:38:47.343ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
134092025-04-30T06:38:47.344ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a2f01e2-a3c2-48ce-979f-fe6a9181352f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134102025-04-30T06:38:47.344ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
134112025-04-30T06:38:47.346ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b3dd760-dd14-4105-a312-3f489164ebf8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134122025-04-30T06:38:47.346ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
134132025-04-30T06:38:47.348ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22a0ccf6-e9c5-4dd2-a30f-fb0469a7fdeb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134142025-04-30T06:38:47.348ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
134152025-04-30T06:38:47.350ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53e3a922-b433-4804-ae11-a8d466201f72", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134162025-04-30T06:38:47.350ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
134172025-04-30T06:38:47.351ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b4d434b-c163-4c83-91b6-31774014110d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134182025-04-30T06:38:47.352ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
134192025-04-30T06:38:47.354ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02e500e3-590f-4c20-9607-68b82854c883", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134202025-04-30T06:38:47.354ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
134212025-04-30T06:38:47.358ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cad6fa4d-3796-464e-8517-7ab0b51c0198", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134222025-04-30T06:38:47.358ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
134232025-04-30T06:38:47.360ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134242025-04-30T06:38:47.360ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134252025-04-30T06:38:47.360ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134262025-04-30T06:38:47.360ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
134272025-04-30T06:38:47.361ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134282025-04-30T06:38:47.363ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "023149ad-887c-4c95-8f12-8b18cb6cdee5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134292025-04-30T06:38:47.364ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134302025-04-30T06:38:47.364ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134312025-04-30T06:38:47.364ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134322025-04-30T06:38:47.364ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134332025-04-30T06:38:47.365ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134342025-04-30T06:38:47.365ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134352025-04-30T06:38:47.377ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134362025-04-30T06:38:47.377ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134372025-04-30T06:38:47.377ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
134382025-04-30T06:38:47.379ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134392025-04-30T06:38:47.382ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134402025-04-30T06:38:47.394ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134412025-04-30T06:38:47.394ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
134422025-04-30T06:38:47.394ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134432025-04-30T06:38:47.394ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134442025-04-30T06:38:47.394ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134452025-04-30T06:38:47.395ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134462025-04-30T06:38:47.397ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134472025-04-30T06:38:47.398ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
134482025-04-30T06:38:47.403ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134492025-04-30T06:38:47.403ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134502025-04-30T06:38:47.404ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
134512025-04-30T06:38:47.404ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134522025-04-30T06:38:47.404ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134532025-04-30T06:38:47.404ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134542025-04-30T06:38:47.478ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
134552025-04-30T06:38:47.478ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
134562025-04-30T06:38:47.479ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:48725 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
134572025-04-30T06:38:47.480ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:48725 req_id = 4319a292-2f41-4459-ac88-4e32427c77e7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
134582025-04-30T06:38:47.484ZINFOtest_quotas (dropshot): request completed latency_us = 4220 local_addr = [::1]:55655 method = GET remote_addr = [::1]:48725 req_id = 4319a292-2f41-4459-ac88-4e32427c77e7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
134592025-04-30T06:38:47.484ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4319a292-2f41-4459-ac88-4e32427c77e7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
134602025-04-30T06:38:47.487ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
134612025-04-30T06:38:47.487ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
134622025-04-30T06:38:47.487ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
134632025-04-30T06:38:47.489ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
134642025-04-30T06:38:47.489ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134652025-04-30T06:38:47.489ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134662025-04-30T06:38:47.489ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134672025-04-30T06:38:47.490ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134682025-04-30T06:38:47.494ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134692025-04-30T06:38:47.504ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
134702025-04-30T06:38:47.504ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134712025-04-30T06:38:47.505ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134722025-04-30T06:38:47.505ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134732025-04-30T06:38:47.541ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
134742025-04-30T06:38:47.558ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
134752025-04-30T06:38:47.558ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
134762025-04-30T06:38:47.558ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 613.137557ms iteration = 48 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
134772025-04-30T06:38:47.558ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 49 reason = Signaled
134782025-04-30T06:38:47.558ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
134792025-04-30T06:38:47.558ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134802025-04-30T06:38:47.558ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134812025-04-30T06:38:47.558ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134822025-04-30T06:38:47.559ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134832025-04-30T06:38:47.562ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134842025-04-30T06:38:47.570ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
134852025-04-30T06:38:47.571ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134862025-04-30T06:38:47.571ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134872025-04-30T06:38:47.576ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134882025-04-30T06:38:47.580ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
134892025-04-30T06:38:47.580ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
134902025-04-30T06:38:47.583ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ef fd 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51017 req_id = cb30fa57-05c2-44fe-8bbd-58f8c1515cf0
134912025-04-30T06:38:47.584ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61437,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51017 req_id = cb30fa57-05c2-44fe-8bbd-58f8c1515cf0
134922025-04-30T06:38:47.584ZDEBGtest_quotas (store): query key key = _mgs._tcp
134932025-04-30T06:38:47.584ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
134942025-04-30T06:38:47.590ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51017 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cb30fa57-05c2-44fe-8bbd-58f8c1515cf0
134952025-04-30T06:38:47.592ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134962025-04-30T06:38:47.592ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134972025-04-30T06:38:47.592ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134982025-04-30T06:38:47.592ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
134992025-04-30T06:38:47.594ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135002025-04-30T06:38:47.595ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3992, tv_nsec: 393275613 } })
135012025-04-30T06:38:47.596ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
135022025-04-30T06:38:47.597ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 56 47 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e VG..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50057 req_id = 453d45c8-a4ca-4e9a-927b-48bcd01bec92
135032025-04-30T06:38:47.598ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22087,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50057 req_id = 453d45c8-a4ca-4e9a-927b-48bcd01bec92
135042025-04-30T06:38:47.598ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
135052025-04-30T06:38:47.598ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50057 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 453d45c8-a4ca-4e9a-927b-48bcd01bec92
135062025-04-30T06:38:47.657ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
135072025-04-30T06:38:47.657ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135082025-04-30T06:38:47.664ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135092025-04-30T06:38:47.664ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135102025-04-30T06:38:47.664ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
135112025-04-30T06:38:47.666ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135122025-04-30T06:38:47.668ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b3 0c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52563 req_id = 8fbe1de8-a368-4581-b4de-4e0e75103b9b
135132025-04-30T06:38:47.668ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45836,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52563 req_id = 8fbe1de8-a368-4581-b4de-4e0e75103b9b
135142025-04-30T06:38:47.668ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
135152025-04-30T06:38:47.669ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45836, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52563 req_id = 8fbe1de8-a368-4581-b4de-4e0e75103b9b
135162025-04-30T06:38:47.669ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
135172025-04-30T06:38:47.669ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
135182025-04-30T06:38:47.670ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
135192025-04-30T06:38:47.671ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135202025-04-30T06:38:47.671ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
135212025-04-30T06:38:47.672ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135222025-04-30T06:38:47.674ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8cde966-856c-4c2b-a173-c860bd3f0505", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135232025-04-30T06:38:47.675ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
135242025-04-30T06:38:47.676ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fd4ba6d-4871-4ab4-88a3-989b25ea7cd4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135252025-04-30T06:38:47.677ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
135262025-04-30T06:38:47.679ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d0086d2-8dd8-4465-8df3-4409578b0037", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135272025-04-30T06:38:47.680ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
135282025-04-30T06:38:47.684ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff79847b-48dd-49f6-b8b4-22cfe5e0ce67", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135292025-04-30T06:38:47.684ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
135302025-04-30T06:38:47.689ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9989f34-d79b-4586-b544-8b0b277f86bd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135312025-04-30T06:38:47.690ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
135322025-04-30T06:38:47.695ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "602bc42f-802f-442e-861b-d552dd09e9c5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135332025-04-30T06:38:47.695ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
135342025-04-30T06:38:47.700ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bf14ec9-4268-40ab-9baa-7cb871efa787", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135352025-04-30T06:38:47.700ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
135362025-04-30T06:38:47.705ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
135372025-04-30T06:38:47.707ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "034b0684-c073-4163-9b91-66f06efdc676", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135382025-04-30T06:38:47.708ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
135392025-04-30T06:38:47.709ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceaceec2-0180-4f68-9f28-e1eb89b45438", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135402025-04-30T06:38:47.710ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
135412025-04-30T06:38:47.711ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0f7922a-f356-4997-85ad-e4bcdf9d8350", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135422025-04-30T06:38:47.711ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
135432025-04-30T06:38:47.713ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65058f97-71b5-4402-bda2-9da526b5a8ed", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135442025-04-30T06:38:47.713ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
135452025-04-30T06:38:47.715ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f33d614-cb35-4256-b080-7ffbf34d431d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135462025-04-30T06:38:47.715ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
135472025-04-30T06:38:47.716ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1453bdbb-9ab6-4a72-8027-882433d72131", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135482025-04-30T06:38:47.716ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
135492025-04-30T06:38:47.718ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab8a2e8d-cb8b-47ac-b8ab-4f0464ce18d0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135502025-04-30T06:38:47.719ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
135512025-04-30T06:38:47.723ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ba4ac01-d451-487f-ab19-45147a5ab891", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135522025-04-30T06:38:47.723ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
135532025-04-30T06:38:47.725ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d67cc20c-cf5f-4c5c-9e5e-a1bfcbc6e1b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135542025-04-30T06:38:47.725ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
135552025-04-30T06:38:47.727ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77e275ef-450f-4021-ab4b-69df6c3aa0d5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135562025-04-30T06:38:47.727ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
135572025-04-30T06:38:47.729ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55156907-ebc2-4808-98de-3b91fa6e3f21", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135582025-04-30T06:38:47.729ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
135592025-04-30T06:38:47.732ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78303506-95fa-4c11-acb6-e65fed2510a7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135602025-04-30T06:38:47.732ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
135612025-04-30T06:38:47.733ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "797779ee-4196-4d07-92c4-378103e9b012", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135622025-04-30T06:38:47.734ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
135632025-04-30T06:38:47.735ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4524e85b-f391-4739-a69b-d91288867b95", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135642025-04-30T06:38:47.735ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
135652025-04-30T06:38:47.736ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f226119-934f-4a0c-94be-53f48ff6f388", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135662025-04-30T06:38:47.736ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
135672025-04-30T06:38:47.737ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb95cb18-853f-46c9-b967-4b810e46308d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135682025-04-30T06:38:47.738ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
135692025-04-30T06:38:47.739ZINFOtest_quotas (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
135702025-04-30T06:38:47.739ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135712025-04-30T06:38:47.739ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d73cc3d7-fdae-47f6-968b-d3bca6bea948 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135722025-04-30T06:38:47.740ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d73cc3d7-fdae-47f6-968b-d3bca6bea948 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135732025-04-30T06:38:47.740ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d73cc3d7-fdae-47f6-968b-d3bca6bea948 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135742025-04-30T06:38:47.741ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d73cc3d7-fdae-47f6-968b-d3bca6bea948 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135752025-04-30T06:38:47.744ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06386f3c-bee2-47a3-914c-34038de3032c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135762025-04-30T06:38:47.744ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
135772025-04-30T06:38:47.746ZINFOtest_quotas (dropshot_internal): request completed latency_us = 6880 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = d73cc3d7-fdae-47f6-968b-d3bca6bea948 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135782025-04-30T06:38:47.747ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "d73cc3d7-fdae-47f6-968b-d3bca6bea948", "content-length": "405", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135792025-04-30T06:38:47.747ZDEBGtest_quotas (oximeter): client request body = None method = GET uri = http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135802025-04-30T06:38:47.748ZTRACtest_quotas (dropshot_internal): incoming request local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 647cd64e-c68a-486e-b401-9008f49bf3a5 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135812025-04-30T06:38:47.748ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f50c6d1f-9f99-49f8-a4bc-aa9dcf81f1ee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135822025-04-30T06:38:47.748ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
135832025-04-30T06:38:47.748ZTRACtest_quotas (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 647cd64e-c68a-486e-b401-9008f49bf3a5 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135842025-04-30T06:38:47.749ZDEBGtest_quotas (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 647cd64e-c68a-486e-b401-9008f49bf3a5 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135852025-04-30T06:38:47.750ZDEBGtest_quotas (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 647cd64e-c68a-486e-b401-9008f49bf3a5 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135862025-04-30T06:38:47.752ZINFOtest_quotas (dropshot_internal): request completed latency_us = 4613 local_addr = 127.0.0.1:64801 method = GET remote_addr = 127.0.0.1:40669 req_id = 647cd64e-c68a-486e-b401-9008f49bf3a5 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135872025-04-30T06:38:47.753ZDEBGtest_quotas (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:64801/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "647cd64e-c68a-486e-b401-9008f49bf3a5", "content-length": "29", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135882025-04-30T06:38:47.753ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 4678dd50-d1e0-4228-8533-c4090342ff58
135892025-04-30T06:38:47.753ZTRACtest_quotas (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135902025-04-30T06:38:47.753ZINFOtest_quotas (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
135912025-04-30T06:38:47.755ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09073cf8-380f-4d61-b146-9003bb195dec", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135922025-04-30T06:38:47.755ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
135932025-04-30T06:38:47.757ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135942025-04-30T06:38:47.757ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135952025-04-30T06:38:47.757ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135962025-04-30T06:38:47.757ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
135972025-04-30T06:38:47.758ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135982025-04-30T06:38:47.760ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f1a2d02-c530-49a9-9811-123fc142e3dd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
135992025-04-30T06:38:47.760ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
136002025-04-30T06:38:47.760ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136012025-04-30T06:38:47.766ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136022025-04-30T06:38:47.767ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136032025-04-30T06:38:47.767ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
136042025-04-30T06:38:47.767ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136052025-04-30T06:38:47.770ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136062025-04-30T06:38:47.770ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
136072025-04-30T06:38:47.771ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136082025-04-30T06:38:47.773ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0178c5ba-0481-4e76-b9cf-51b18d4b197d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136092025-04-30T06:38:47.773ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
136102025-04-30T06:38:47.775ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd9a6a65-2507-4fc0-9b5c-d2ba5bb61ce1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136112025-04-30T06:38:47.775ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
136122025-04-30T06:38:47.778ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2234f6e-8d24-42d0-9a83-ca1c3b56ee40", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136132025-04-30T06:38:47.778ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
136142025-04-30T06:38:47.780ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "330fdded-585b-4674-832e-dc2bc7176921", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136152025-04-30T06:38:47.780ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
136162025-04-30T06:38:47.781ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13be21ce-e8d8-461c-a195-c03e2209f036", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136172025-04-30T06:38:47.781ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
136182025-04-30T06:38:47.782ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f53152c9-6353-476e-aa50-6e0b72fc557a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136192025-04-30T06:38:47.782ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa
136202025-04-30T06:38:47.783ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36b7aecb-9ccb-41d1-b4c6-4ad9b9f60b53", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136212025-04-30T06:38:47.783ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1
136222025-04-30T06:38:47.784ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87f657a7-a3e3-4b61-9eb7-5d5d66c67696", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136232025-04-30T06:38:47.784ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0
136242025-04-30T06:38:47.786ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ec7d424-f024-417e-9344-e86816bc7435", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136252025-04-30T06:38:47.787ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1
136262025-04-30T06:38:47.789ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47f24acb-0705-41d8-9ad4-e4499f01cb48", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136272025-04-30T06:38:47.789ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0
136282025-04-30T06:38:47.794ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb8edb9c-213a-4cb4-b42a-48668a847fe0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136292025-04-30T06:38:47.795ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1
136302025-04-30T06:38:47.796ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac46d51f-045d-4029-8812-550b48ec1163", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136312025-04-30T06:38:47.796ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0
136322025-04-30T06:38:47.801ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
136332025-04-30T06:38:47.803ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d56fbe54-678d-4328-8b87-155c987d5ecb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136342025-04-30T06:38:47.804ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1
136352025-04-30T06:38:47.806ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c603724-0462-4a38-879f-f3df492b6e9d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136362025-04-30T06:38:47.806ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cmpa
136372025-04-30T06:38:47.815ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37159554-7936-4533-ad0b-8120a607ed91", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136382025-04-30T06:38:47.815ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
136392025-04-30T06:38:47.817ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcd6d150-330a-4459-a78a-2c8e2b0d690e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136402025-04-30T06:38:47.817ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
136412025-04-30T06:38:47.818ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7972f39-03ec-4b96-b4c2-9bab6c9c9427", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136422025-04-30T06:38:47.819ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/1/component/rot/cfpa
136432025-04-30T06:38:47.820ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdb862d5-ed98-42b0-886b-8eebdd2e4335", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136442025-04-30T06:38:47.820ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136452025-04-30T06:38:47.820ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136462025-04-30T06:38:47.820ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136472025-04-30T06:38:47.820ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136482025-04-30T06:38:47.823ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136492025-04-30T06:38:47.841ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136502025-04-30T06:38:47.846ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
136512025-04-30T06:38:47.846ZTRACtest_quotas (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
136522025-04-30T06:38:47.846ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136532025-04-30T06:38:47.846ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136542025-04-30T06:38:47.846ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136552025-04-30T06:38:47.847ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136562025-04-30T06:38:47.851ZDEBGtest_quotas (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136572025-04-30T06:38:47.851ZDEBGtest_quotas (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136582025-04-30T06:38:47.851ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136592025-04-30T06:38:47.855ZDEBGtest_quotas (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
136602025-04-30T06:38:47.856ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136612025-04-30T06:38:47.856ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136622025-04-30T06:38:47.856ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136632025-04-30T06:38:47.857ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136642025-04-30T06:38:47.857ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136652025-04-30T06:38:47.857ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136662025-04-30T06:38:47.857ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
136672025-04-30T06:38:47.860ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136682025-04-30T06:38:47.923ZDEBGtest_quotas (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:55655
136692025-04-30T06:38:47.923ZDEBGtest_quotas (BackgroundTasks): client request SledAgent = http://[::1]:55655 background_task = inventory_collection body = None method = GET uri = http://[::1]:55655/inventory
136702025-04-30T06:38:47.925ZINFOtest_quotas (dropshot): accepted connection local_addr = [::1]:55655 remote_addr = [::1]:36888 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
136712025-04-30T06:38:47.925ZTRACtest_quotas (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136722025-04-30T06:38:47.926ZDEBGtest_quotas (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136732025-04-30T06:38:47.926ZTRACtest_quotas (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136742025-04-30T06:38:47.926ZTRACtest_quotas (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136752025-04-30T06:38:47.934ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136762025-04-30T06:38:47.942ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136772025-04-30T06:38:47.942ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136782025-04-30T06:38:47.942ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
136792025-04-30T06:38:47.943ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136802025-04-30T06:38:47.944ZTRACtest_quotas (dropshot): incoming request local_addr = [::1]:55655 method = GET remote_addr = [::1]:36888 req_id = 30a33d96-b75f-436c-8e22-38ace258ba94 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
136812025-04-30T06:38:47.945ZINFOtest_quotas (dropshot): request completed latency_us = 1019 local_addr = [::1]:55655 method = GET remote_addr = [::1]:36888 req_id = 30a33d96-b75f-436c-8e22-38ace258ba94 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
136822025-04-30T06:38:47.950ZDEBGtest_quotas (BackgroundTasks): client response SledAgent = http://[::1]:55655 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:55655/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "30a33d96-b75f-436c-8e22-38ace258ba94", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
136832025-04-30T06:38:47.953ZDEBGtest_quotas (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
136842025-04-30T06:38:47.953ZDEBGtest_quotas (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
136852025-04-30T06:38:47.953ZDEBGtest_quotas (BackgroundTasks): finished collection background_task = inventory_collection
136862025-04-30T06:38:47.956ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
136872025-04-30T06:38:47.956ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136882025-04-30T06:38:47.956ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136892025-04-30T06:38:47.956ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136902025-04-30T06:38:47.956ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136912025-04-30T06:38:47.959ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136922025-04-30T06:38:47.959ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
136932025-04-30T06:38:47.961ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136942025-04-30T06:38:47.966ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136952025-04-30T06:38:47.972ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
136962025-04-30T06:38:47.972ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136972025-04-30T06:38:47.972ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136982025-04-30T06:38:47.973ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136992025-04-30T06:38:48.032ZWARNtest_quotas (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
137002025-04-30T06:38:48.032ZWARNtest_quotas (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
137012025-04-30T06:38:48.032ZDEBGtest_quotas (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 474.173139ms iteration = 49 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
137022025-04-30T06:38:48.033ZDEBGtest_quotas (BackgroundTasks): activating background_task = inventory_collection iteration = 50 reason = Signaled
137032025-04-30T06:38:48.033ZTRACtest_quotas (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
137042025-04-30T06:38:48.035ZTRACtest_quotas (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137052025-04-30T06:38:48.035ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137062025-04-30T06:38:48.035ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137072025-04-30T06:38:48.037ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137082025-04-30T06:38:48.050ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
137092025-04-30T06:38:48.060ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137102025-04-30T06:38:48.065ZDEBGtest_quotas (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
137112025-04-30T06:38:48.065ZTRACtest_quotas (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137122025-04-30T06:38:48.065ZDEBGtest_quotas (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137132025-04-30T06:38:48.066ZDEBGtest_quotas (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137142025-04-30T06:38:48.069ZDEBGtest_quotas (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(c920987f-1d0e-4cd0-91b6-107a19a5a911, 0)]
137152025-04-30T06:38:48.069ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
137162025-04-30T06:38:48.070ZTRACtest_quotas (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 56 97 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 V............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57180 req_id = 60b20875-889d-43a2-a5ea-06a9b2698c59
137172025-04-30T06:38:48.070ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22167,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57180 req_id = 60b20875-889d-43a2-a5ea-06a9b2698c59
137182025-04-30T06:38:48.071ZDEBGtest_quotas (store): query key key = _mgs._tcp
137192025-04-30T06:38:48.071ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137202025-04-30T06:38:48.071ZDEBGtest_quotas (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57180 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 60b20875-889d-43a2-a5ea-06a9b2698c59
137212025-04-30T06:38:48.072ZDEBGtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 56554, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3992, tv_nsec: 871238764 } })
137222025-04-30T06:38:48.072ZTRACtest_quotas (DnsResolver): lookup_service_targets: looking up SRV target
137232025-04-30T06:38:48.073ZTRACtest_quotas (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fd 13 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54398 req_id = f3c2ca0c-b898-4d7a-8a71-103c8650b114
137242025-04-30T06:38:48.073ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64787,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54398 req_id = f3c2ca0c-b898-4d7a-8a71-103c8650b114
137252025-04-30T06:38:48.073ZDEBGtest_quotas (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
137262025-04-30T06:38:48.073ZDEBGtest_quotas (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54398 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f3c2ca0c-b898-4d7a-8a71-103c8650b114
137272025-04-30T06:38:48.118ZTRACtest_quotas (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
137282025-04-30T06:38:48.119ZTRACtest_quotas (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137292025-04-30T06:38:48.119ZTRACtest_quotas (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137302025-04-30T06:38:48.119ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137312025-04-30T06:38:48.119ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
137322025-04-30T06:38:48.120ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137332025-04-30T06:38:48.120ZDEBGtest_quotas (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137342025-04-30T06:38:48.120ZDEBGtest_quotas (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137352025-04-30T06:38:48.121ZDEBGtest_quotas (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b
137362025-04-30T06:38:48.122ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
137372025-04-30T06:38:48.122ZTRACtest_quotas (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: bc a0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59455 req_id = 8c3f5e4d-4535-4e1e-bbe8-0cbe86a84f39
137382025-04-30T06:38:48.122ZDEBGtest_quotas (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48288,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59455 req_id = 8c3f5e4d-4535-4e1e-bbe8-0cbe86a84f39
137392025-04-30T06:38:48.123ZDEBGtest_quotas (store): query key key = _clickhouse-admin-keeper._tcp
137402025-04-30T06:38:48.123ZERROtest_quotas (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48288, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59455 req_id = 8c3f5e4d-4535-4e1e-bbe8-0cbe86a84f39
137412025-04-30T06:38:48.123ZDEBGtest_quotas (BackgroundTasks): begin collection background_task = inventory_collection
137422025-04-30T06:38:48.123ZDEBGtest_quotas (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:56554
137432025-04-30T06:38:48.123ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/ignition
137442025-04-30T06:38:48.124ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137452025-04-30T06:38:48.129ZDEBGtest_quotas (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137462025-04-30T06:38:48.130ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137472025-04-30T06:38:48.130ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
137482025-04-30T06:38:48.130ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137492025-04-30T06:38:48.133ZTRACtest_quotas (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137502025-04-30T06:38:48.133ZDEBGtest_quotas (InternalApi): roles roles = RoleSet { roles: {} }
137512025-04-30T06:38:48.134ZDEBGtest_quotas (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137522025-04-30T06:38:48.135ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 2 table_name = fields_string
137532025-04-30T06:38:48.135ZTRACtest_quotas (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
137542025-04-30T06:38:48.137ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc15229e-6800-4ed6-b7cb-d93cd5ec527d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137552025-04-30T06:38:48.137ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0
137562025-04-30T06:38:48.140ZDEBGtest_quotas (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = a24f3fbe-6c58-493e-9445-8078333aa41b n_rows = 1 table_name = measurements_i64
137572025-04-30T06:38:48.140ZTRACtest_quotas (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
137582025-04-30T06:38:48.142ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40069327-2cb5-4450-984f-db58bfa0a60c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137592025-04-30T06:38:48.143ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0
137602025-04-30T06:38:48.147ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77dd8a92-1a47-406d-86cb-e0b94975290b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137612025-04-30T06:38:48.147ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1
137622025-04-30T06:38:48.154ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "388afbaa-298e-4839-b1ac-3887d40b1365", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137632025-04-30T06:38:48.154ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0
137642025-04-30T06:38:48.157ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cd34af9-a741-49b2-82a7-a09b3a2777dd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137652025-04-30T06:38:48.157ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1
137662025-04-30T06:38:48.160ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47ea8bd1-45ae-492f-a540-0cc8e8e3093b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137672025-04-30T06:38:48.160ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0
137682025-04-30T06:38:48.162ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d38a94f-6ae1-4f4f-8940-40b429f9b646", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137692025-04-30T06:38:48.163ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1
137702025-04-30T06:38:48.167ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c34f5691-c140-4d91-b24b-8184185d727d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137712025-04-30T06:38:48.167ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cmpa
137722025-04-30T06:38:48.167ZDEBGtest_quotas: attempt to fetch latest inventory collection result = Ok("found")
137732025-04-30T06:38:48.169ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34a7a99f-e45e-466a-b311-829df49fdb93", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137742025-04-30T06:38:48.169ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
137752025-04-30T06:38:48.170ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6de02e35-c8a6-455d-b5be-2907fe90c161", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137762025-04-30T06:38:48.170ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
137772025-04-30T06:38:48.171ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1b06804-5825-4a12-86ae-1f275a093632", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137782025-04-30T06:38:48.171ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/0/component/rot/cfpa
137792025-04-30T06:38:48.172ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0a53f03-bf7b-4e45-9e4d-f9bbde430f59", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137802025-04-30T06:38:48.173ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1
137812025-04-30T06:38:48.174ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8271361a-02b7-4a58-ab4a-97d9f1bc358f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137822025-04-30T06:38:48.174ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0
137832025-04-30T06:38:48.176ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12945c4e-62bb-447d-9d91-d507b50a6d8f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137842025-04-30T06:38:48.176ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1
137852025-04-30T06:38:48.181ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "069242a0-61fc-4153-b686-5d98af610ca5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137862025-04-30T06:38:48.181ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0
137872025-04-30T06:38:48.183ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11ed24d9-e16e-4af9-8f17-b9650d57c36f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137882025-04-30T06:38:48.183ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1
137892025-04-30T06:38:48.185ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61861add-249e-4198-a55e-15623a9449cf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137902025-04-30T06:38:48.185ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0
137912025-04-30T06:38:48.188ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec03448c-7046-4093-a22d-e1dcc5b075d4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137922025-04-30T06:38:48.188ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1
137932025-04-30T06:38:48.191ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da8c0ab7-f92b-434b-bbf0-9093b3e7537d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137942025-04-30T06:38:48.191ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cmpa
137952025-04-30T06:38:48.192ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d524422-83b1-40b6-ae86-bb7f5b03939b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137962025-04-30T06:38:48.192ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
137972025-04-30T06:38:48.195ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b31c7452-cb2e-4779-9126-1b6347cf0248", "content-length": "727", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
137982025-04-30T06:38:48.195ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
137992025-04-30T06:38:48.196ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f758c3b-070d-4c04-989f-50302c29f11d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
138002025-04-30T06:38:48.196ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/switch/1/component/rot/cfpa
138012025-04-30T06:38:48.198ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7834305f-fe23-4fac-aead-12424da17835", "content-length": "728", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
138022025-04-30T06:38:48.198ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0
138032025-04-30T06:38:48.200ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27ab56c9-f5d2-466b-898b-042e8f31f02b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
138042025-04-30T06:38:48.201ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0
138052025-04-30T06:38:48.203ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ec57673-ba93-471a-b79f-335df5b52f93", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
138062025-04-30T06:38:48.203ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1
138072025-04-30T06:38:48.205ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03cc487d-0269-4932-989c-7972a241f752", "content-length": "109", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
138082025-04-30T06:38:48.205ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0
138092025-04-30T06:38:48.207ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "626bb2cf-a8e9-481a-9ecd-aaf9b532b3fa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
138102025-04-30T06:38:48.207ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1
138112025-04-30T06:38:48.209ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba5778ba-e931-4b79-be0b-b3e310d789e6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
138122025-04-30T06:38:48.210ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0
138132025-04-30T06:38:48.213ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd062d6c-fb97-4480-a342-8646606bebb6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
138142025-04-30T06:38:48.213ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1
138152025-04-30T06:38:48.215ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3dddf55-baad-4fb3-8d81-20681faa7d8b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
138162025-04-30T06:38:48.216ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cmpa
138172025-04-30T06:38:48.217ZDEBGtest_quotas (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:56554 result = Ok(Response { url: "http://[::1]:56554/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e78a17a-6d4a-4c1c-af96-6ee4327714bb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:38:47 GMT"} })
138182025-04-30T06:38:48.217ZDEBGtest_quotas (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:56554 method = GET uri = http://[::1]:56554/sp/sled/0/component/rot/cfpa